[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:43:32 dist-test-slave-n563 krb5kdc[5353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:43:32 dist-test-slave-n563 krb5kdc[5353](info): set up 2 sockets
Sep 25 02:43:32 dist-test-slave-n563 krb5kdc[5353](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 02:43:33.944657 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.006s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:43:34 dist-test-slave-n563 krb5kdc[5353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:43:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:43:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:43:34.134088 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43365
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:43365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:34.515144 5366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:34.515514 5366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:34.515610 5366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:34.523380 5366 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:43:34.523510 5366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:34.523566 5366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:34.523626 5366 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:43:34.523672 5366 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:43:34.534858 5366 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:43365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:34.536585 5366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:34.538221 5366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:34.554672 5375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:34.555285 5373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:34.555299 5372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:34.555859 5366 server_base.cc:1034] running on GCE node
I20260925 02:43:34.556924 5366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:34.558939 5366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:34.560153 5366 hybrid_clock.cc:648] HybridClock initialized: now 1790304214560112 us; error 35 us; skew 500 ppm
I20260925 02:43:34.560563 5366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:34 dist-test-slave-n563 krb5kdc[5353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304214, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:34.564003 5366 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:43:34.566187 5366 webserver.cc:460] Webserver started at http://127.5.56.190:46327/ using document root <none> and password file <none>
I20260925 02:43:34.567095 5366 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:34.567232 5366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:34.567643 5366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:34.571455 5366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "3d01f659585f4d5ea39056e27b9779a2"
format_stamp: "Formatted at 2026-09-25 02:43:34 on dist-test-slave-n563"
server_key: "5149b1e50c404a2f427316c1b51d0f17"
server_key_iv: "cc3eb69a368ab07554e6e4b516c0a8ea"
server_key_version: "encryptionkey@0"
I20260925 02:43:34.572257 5366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "3d01f659585f4d5ea39056e27b9779a2"
format_stamp: "Formatted at 2026-09-25 02:43:34 on dist-test-slave-n563"
server_key: "5149b1e50c404a2f427316c1b51d0f17"
server_key_iv: "cc3eb69a368ab07554e6e4b516c0a8ea"
server_key_version: "encryptionkey@0"
I20260925 02:43:34.578845 5366 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260925 02:43:34.583272 5382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:34.585373 5366 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:43:34.585567 5366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "3d01f659585f4d5ea39056e27b9779a2"
format_stamp: "Formatted at 2026-09-25 02:43:34 on dist-test-slave-n563"
server_key: "5149b1e50c404a2f427316c1b51d0f17"
server_key_iv: "cc3eb69a368ab07554e6e4b516c0a8ea"
server_key_version: "encryptionkey@0"
I20260925 02:43:34.585791 5366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:34.616184 5366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:34.621381 5366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:34.621778 5366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:34.642342 5366 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:43365
I20260925 02:43:34.642367 5433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:43365 every 8 connection(s)
I20260925 02:43:34.644220 5366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:43:34.647594 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 5366
I20260925 02:43:34.647930 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:43:34.649756 5434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:34.650352 5346 external_mini_cluster.cc:1428] Setting key 7b639bcf266a600568593ceb9f37253d
I20260925 02:43:34.664628 5434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2: Bootstrap starting.
I20260925 02:43:34.669324 5434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:34.670917 5434 log.cc:824] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:34.674600 5434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2: No bootstrap required, opened a new log
Sep 25 02:43:34 dist-test-slave-n563 krb5kdc[5353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304214, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:34.685684 5434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d01f659585f4d5ea39056e27b9779a2" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43365 } }
I20260925 02:43:34.686329 5434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:34.686537 5434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d01f659585f4d5ea39056e27b9779a2, State: Initialized, Role: FOLLOWER
I20260925 02:43:34.687680 5434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d01f659585f4d5ea39056e27b9779a2" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43365 } }
I20260925 02:43:34.688097 5434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:43:34.688342 5434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:43:34.688572 5434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:34.691847 5434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d01f659585f4d5ea39056e27b9779a2" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43365 } }
I20260925 02:43:34.692970 5434 leader_election.cc:302] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d01f659585f4d5ea39056e27b9779a2; no voters:
I20260925 02:43:34.693197 5436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:34.655505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60544 (local address 127.5.56.190:43365)
0925 02:43:34.656558 (+ 1053us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:34.656575 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:34.657732 (+ 1157us) server_negotiation.cc:407] Connection header received
0925 02:43:34.666882 (+ 9150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:34.666909 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:34.667392 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:34.667800 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:34.671154 (+ 3354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:34.672995 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:34.674829 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:34.676152 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:34.676445 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:34.682291 (+ 5846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:34.682320 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:34.682344 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:34.682467 (+ 123us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:34.686947 (+ 4480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:34.687989 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:34.687999 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:34.688007 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:34.688303 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:34.688911 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:34.688918 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:34.688922 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:34.689875 (+ 953us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:34.690259 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:34.691225 (+ 966us) server_negotiation.cc:299] Negotiation successful
0925 02:43:34.691697 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":714,"thread_start_us":416,"threads_started":1}
I20260925 02:43:34.693914 5434 leader_election.cc:288] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:43:34.694123 5439 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:34.694856 5439 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [term 1 LEADER]: Becoming Leader. State: Replica: 3d01f659585f4d5ea39056e27b9779a2, State: Running, Role: LEADER
I20260925 02:43:34.695858 5439 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d01f659585f4d5ea39056e27b9779a2" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43365 } }
I20260925 02:43:34.697130 5434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:43:34.700034 5441 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d01f659585f4d5ea39056e27b9779a2. Latest consensus state: current_term: 1 leader_uuid: "3d01f659585f4d5ea39056e27b9779a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d01f659585f4d5ea39056e27b9779a2" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43365 } } }
I20260925 02:43:34.700541 5441 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:43:34.700176 5442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d01f659585f4d5ea39056e27b9779a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d01f659585f4d5ea39056e27b9779a2" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43365 } } }
I20260925 02:43:34.700945 5442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2 [sys.catalog]: This master's current role is: LEADER
I20260925 02:43:34.704597 5445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:43:34.711460 5445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:43:34.724542 5445 catalog_manager.cc:1347] Generated new cluster ID: 27a29330b1304f17a7e1a4c9bb5751ce
I20260925 02:43:34.724673 5445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:43:34.764247 5445 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:43:34.765690 5445 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:43:34.784724 5445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d01f659585f4d5ea39056e27b9779a2: Generated new TSK 0
I20260925 02:43:34.785770 5445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:43:34.831367 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43365
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:35.179407 5461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:35.179878 5461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:35.180052 5461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:35.188084 5461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:35.188226 5461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:35.188403 5461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:43:35.199314 5461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:35.200681 5461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:35.202054 5461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:35.216125 5466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:35.216090 5467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:35.216094 5469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:35.217034 5461 server_base.cc:1034] running on GCE node
I20260925 02:43:35.217736 5461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:35.218824 5461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:35.220052 5461 hybrid_clock.cc:648] HybridClock initialized: now 1790304215219992 us; error 54 us; skew 500 ppm
I20260925 02:43:35.220384 5461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:35 dist-test-slave-n563 krb5kdc[5353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304215, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:35.223699 5461 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:43:35.225138 5461 webserver.cc:460] Webserver started at http://127.5.56.129:45025/ using document root <none> and password file <none>
I20260925 02:43:35.225943 5461 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:35.226229 5461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:35.226691 5461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:35.231297 5461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "33f755196e0b447281f702381d503c5c"
format_stamp: "Formatted at 2026-09-25 02:43:35 on dist-test-slave-n563"
server_key: "870b415f8d3674f4c3d42f6cf89186ab"
server_key_iv: "d10092983e03a92abe4915f92df98187"
server_key_version: "encryptionkey@0"
I20260925 02:43:35.232532 5461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "33f755196e0b447281f702381d503c5c"
format_stamp: "Formatted at 2026-09-25 02:43:35 on dist-test-slave-n563"
server_key: "870b415f8d3674f4c3d42f6cf89186ab"
server_key_iv: "d10092983e03a92abe4915f92df98187"
server_key_version: "encryptionkey@0"
I20260925 02:43:35.238513 5461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 02:43:35.242081 5476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:35.243512 5461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:43:35.243718 5461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "33f755196e0b447281f702381d503c5c"
format_stamp: "Formatted at 2026-09-25 02:43:35 on dist-test-slave-n563"
server_key: "870b415f8d3674f4c3d42f6cf89186ab"
server_key_iv: "d10092983e03a92abe4915f92df98187"
server_key_version: "encryptionkey@0"
I20260925 02:43:35.243934 5461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:35.261641 5461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:35.266667 5461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:35.266990 5461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:35.268100 5461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:43:35.269811 5461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:35.269917 5461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:35.270083 5461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:35.270356 5461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:35.307632 5461 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:35829
I20260925 02:43:35.307688 5589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:35829 every 8 connection(s)
I20260925 02:43:35.309016 5461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:43:35.309693 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 5461
I20260925 02:43:35.309934 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:43:35.310817 5346 external_mini_cluster.cc:1428] Setting key ad216b75a71c5edee9fe0546d2bbac81
Sep 25 02:43:35 dist-test-slave-n563 krb5kdc[5353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304215, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
I20260925 02:43:35.331451 5592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:35.312423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:42991 (local address 127.5.56.190:43365)
0925 02:43:35.313166 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:35.313184 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:35.314509 (+ 1325us) server_negotiation.cc:407] Connection header received
0925 02:43:35.316427 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:35.316433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:35.316519 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:35.316657 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:35.318205 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:35.319020 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:35.320792 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:35.321220 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:35.321367 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:35.325007 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:35.325012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:35.325015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:35.325050 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:35.327916 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:35.328710 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:35.328716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:35.328720 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:35.328884 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:35.329397 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:35.329400 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:35.329402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:35.329774 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:35.329942 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:35.330812 (+ 870us) server_negotiation.cc:299] Negotiation successful
0925 02:43:35.331052 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":315,"threads_started":1}
I20260925 02:43:35.332500 5593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:35.313264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43365 (local address 127.5.56.129:42991)
0925 02:43:35.314284 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:35.314330 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:35.316157 (+ 1827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:35.316909 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:35.316918 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:35.317444 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:35.318027 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:35.318044 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:35.319180 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:35.319193 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:35.320636 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:35.320648 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:35.321466 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:35.321475 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:35.321738 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:35.322516 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:35.322544 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:35.324762 (+ 2218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:35.328061 (+ 3299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:35.328070 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:35.328090 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:35.328541 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:35.329069 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:35.329073 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:35.329076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:35.329283 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:35.329905 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:35.329916 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:35.330295 (+ 379us) client_negotiation.cc:769] Sending connection context
0925 02:43:35.330715 (+ 420us) client_negotiation.cc:240] Negotiation successful
0925 02:43:35.331288 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":398,"threads_started":1}
I20260925 02:43:35.334290 5590 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43365
I20260925 02:43:35.334714 5590 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:35.335764 5590 heartbeater.cc:511] Master 127.5.56.190:43365 requested a full tablet report, sending...
I20260925 02:43:35.338691 5399 ts_manager.cc:194] Registered new tserver with Master: 33f755196e0b447281f702381d503c5c (127.5.56.129:35829)
I20260925 02:43:35.340559 5399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:42991
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:43:35.354658 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43365
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:35.705546 5596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:35.705835 5596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:35.705936 5596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:35.713488 5596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:35.713613 5596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:35.713732 5596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:43:35.723737 5596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:35.725241 5596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:35.726747 5596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:35.740530 5601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:35.740689 5604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:35.740741 5602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:35.741852 5596 server_base.cc:1034] running on GCE node
I20260925 02:43:35.742600 5596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:35.743546 5596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:35.744751 5596 hybrid_clock.cc:648] HybridClock initialized: now 1790304215744682 us; error 59 us; skew 500 ppm
I20260925 02:43:35.745051 5596 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:35 dist-test-slave-n563 krb5kdc[5353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304215, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:35.748867 5596 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:43:35.750471 5596 webserver.cc:460] Webserver started at http://127.5.56.130:37033/ using document root <none> and password file <none>
I20260925 02:43:35.751174 5596 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:35.751281 5596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:35.751555 5596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:35.754891 5596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "da5e718652ff4754b4f8ed267b03c166"
format_stamp: "Formatted at 2026-09-25 02:43:35 on dist-test-slave-n563"
server_key: "7e4f830def631d44790df0d78a78055c"
server_key_iv: "981805f11afa05c84eb4819a36bf28be"
server_key_version: "encryptionkey@0"
I20260925 02:43:35.755686 5596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "da5e718652ff4754b4f8ed267b03c166"
format_stamp: "Formatted at 2026-09-25 02:43:35 on dist-test-slave-n563"
server_key: "7e4f830def631d44790df0d78a78055c"
server_key_iv: "981805f11afa05c84eb4819a36bf28be"
server_key_version: "encryptionkey@0"
I20260925 02:43:35.761538 5596 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:43:35.765516 5611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:35.767099 5596 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:43:35.767287 5596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "da5e718652ff4754b4f8ed267b03c166"
format_stamp: "Formatted at 2026-09-25 02:43:35 on dist-test-slave-n563"
server_key: "7e4f830def631d44790df0d78a78055c"
server_key_iv: "981805f11afa05c84eb4819a36bf28be"
server_key_version: "encryptionkey@0"
I20260925 02:43:35.767549 5596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:35.789569 5596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:35.794607 5596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:35.794922 5596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:35.795972 5596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:43:35.797521 5596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:35.797598 5596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:35.797771 5596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:35.797823 5596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:35.833834 5596 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:37087
I20260925 02:43:35.833894 5724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:37087 every 8 connection(s)
I20260925 02:43:35.835216 5596 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:43:35.843886 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 5596
I20260925 02:43:35.844153 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:43:35.844734 5346 external_mini_cluster.cc:1428] Setting key 5465a927c549376e5327dafda0522f76
Sep 25 02:43:35 dist-test-slave-n563 krb5kdc[5353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304215, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:35.857589 5727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:35.837931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:38221 (local address 127.5.56.190:43365)
0925 02:43:35.838358 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:35.838363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:35.839509 (+ 1146us) server_negotiation.cc:407] Connection header received
0925 02:43:35.841369 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:35.841375 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:35.841466 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:35.841600 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:35.843210 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:35.843937 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:35.846211 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:35.846895 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:35.847052 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:35.851597 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:35.851602 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:35.851606 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:35.851646 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:35.854347 (+ 2701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:35.855208 (+ 861us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:35.855218 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:35.855222 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:35.855349 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:35.855800 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:35.855805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:35.855807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:35.856073 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:35.856228 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:35.856973 (+ 745us) server_negotiation.cc:299] Negotiation successful
0925 02:43:35.857207 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":219,"threads_started":1}
I20260925 02:43:35.858716 5728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:35.838614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43365 (local address 127.5.56.130:38221)
0925 02:43:35.839343 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:35.839393 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:35.841098 (+ 1705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:35.841845 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:35.841854 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:35.842380 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:35.843016 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:35.843033 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:35.844190 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:35.844198 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:35.846066 (+ 1868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:35.846077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:35.847156 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:35.847163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:35.847467 (+ 304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:35.848330 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:35.848367 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:35.851334 (+ 2967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:35.854567 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:35.854577 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:35.854597 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:35.855062 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:35.855461 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:35.855466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:35.855469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:35.855674 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:35.856202 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:35.856212 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:35.856599 (+ 387us) client_negotiation.cc:769] Sending connection context
0925 02:43:35.856877 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 02:43:35.857466 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":290,"threads_started":1}
I20260925 02:43:35.860241 5725 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43365
I20260925 02:43:35.860627 5725 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:35.861635 5725 heartbeater.cc:511] Master 127.5.56.190:43365 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
I20260925 02:43:35.863454 5399 ts_manager.cc:194] Registered new tserver with Master: da5e718652ff4754b4f8ed267b03c166 (127.5.56.130:37087)
I20260925 02:43:35.864286 5399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:38221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:43:35.887591 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43365
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36201
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:36.234845 5731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:36.235143 5731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:36.235241 5731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:36.242933 5731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:36.243069 5731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:36.243198 5731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:43:36.253501 5731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36201
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43365
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:36.254963 5731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:36.256268 5731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:36.270429 5737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:36.270361 5736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:36.270491 5739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:36.271341 5731 server_base.cc:1034] running on GCE node
I20260925 02:43:36.272001 5731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:36.273034 5731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:36.274251 5731 hybrid_clock.cc:648] HybridClock initialized: now 1790304216274206 us; error 39 us; skew 500 ppm
I20260925 02:43:36.274608 5731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:36 dist-test-slave-n563 krb5kdc[5353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304216, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:36.277757 5731 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:43:36.279381 5731 webserver.cc:460] Webserver started at http://127.5.56.131:40027/ using document root <none> and password file <none>
I20260925 02:43:36.280112 5731 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:36.280246 5731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:36.280582 5731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:36.283624 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "44b94a8af11048e7bbb893d45570139a"
format_stamp: "Formatted at 2026-09-25 02:43:36 on dist-test-slave-n563"
server_key: "edf5b06a725c8c48e0c5548061ae2b4c"
server_key_iv: "2605d648591e70db5acff6b1aa7d66ae"
server_key_version: "encryptionkey@0"
I20260925 02:43:36.284406 5731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "44b94a8af11048e7bbb893d45570139a"
format_stamp: "Formatted at 2026-09-25 02:43:36 on dist-test-slave-n563"
server_key: "edf5b06a725c8c48e0c5548061ae2b4c"
server_key_iv: "2605d648591e70db5acff6b1aa7d66ae"
server_key_version: "encryptionkey@0"
I20260925 02:43:36.289742 5731 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:43:36.293326 5746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:36.294754 5731 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:43:36.294931 5731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "44b94a8af11048e7bbb893d45570139a"
format_stamp: "Formatted at 2026-09-25 02:43:36 on dist-test-slave-n563"
server_key: "edf5b06a725c8c48e0c5548061ae2b4c"
server_key_iv: "2605d648591e70db5acff6b1aa7d66ae"
server_key_version: "encryptionkey@0"
I20260925 02:43:36.295135 5731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:36.314353 5731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:36.318372 5731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:36.318735 5731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:36.319731 5731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:43:36.321123 5731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:36.321195 5731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:36.321372 5731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:36.321422 5731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:36.344167 5590 heartbeater.cc:503] Master 127.5.56.190:43365 was elected leader, sending a full tablet report...
I20260925 02:43:36.357522 5731 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:40061
I20260925 02:43:36.357564 5859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:40061 every 8 connection(s)
I20260925 02:43:36.358982 5731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:43:36.366780 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 5731
I20260925 02:43:36.367008 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:43:36.367486 5346 external_mini_cluster.cc:1428] Setting key c7df9a405876a662caef7eaa4b840166
Sep 25 02:43:36 dist-test-slave-n563 krb5kdc[5353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304216, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:36.380517 5862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:36.361778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:60153 (local address 127.5.56.190:43365)
0925 02:43:36.362237 (+ 459us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:36.362244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:36.363358 (+ 1114us) server_negotiation.cc:407] Connection header received
0925 02:43:36.365155 (+ 1797us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:36.365159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:36.365240 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:36.365331 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:36.367198 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.367994 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.369981 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.370469 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.370601 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:36.374568 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:36.374573 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:36.374577 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:36.374611 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:36.377372 (+ 2761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.378165 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.378169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.378171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.378286 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.378872 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.378877 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.378879 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.379173 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:36.379297 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:36.380071 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 02:43:36.380241 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":199,"threads_started":1}
I20260925 02:43:36.381511 5863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:36.362458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43365 (local address 127.5.56.131:60153)
0925 02:43:36.363188 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:36.363232 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:36.364885 (+ 1653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:36.365644 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:36.365652 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:36.366163 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:36.366987 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:36.367015 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.368201 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.368206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:36.369814 (+ 1608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:36.369826 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.370692 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.370701 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:36.370952 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:36.371756 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:36.371789 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:36.374258 (+ 2469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:36.377507 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:36.377513 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:36.377526 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:36.378040 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:36.378505 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:36.378509 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:36.378512 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:36.378719 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:36.379319 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:36.379325 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:36.379678 (+ 353us) client_negotiation.cc:769] Sending connection context
0925 02:43:36.379924 (+ 246us) client_negotiation.cc:240] Negotiation successful
0925 02:43:36.380320 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":275,"threads_started":1}
I20260925 02:43:36.383188 5860 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43365
I20260925 02:43:36.383515 5860 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:36.384382 5860 heartbeater.cc:511] Master 127.5.56.190:43365 requested a full tablet report, sending...
I20260925 02:43:36.386159 5399 ts_manager.cc:194] Registered new tserver with Master: 44b94a8af11048e7bbb893d45570139a (127.5.56.131:40061)
I20260925 02:43:36.387266 5399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:60153
I20260925 02:43:36.397137 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:43:36 dist-test-slave-n563 krb5kdc[5353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304216, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:43:36 dist-test-slave-n563 krb5kdc[5353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304216, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:43:36.445351 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:36.428071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.5.56.129:35829)
0925 02:43:36.428740 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:36.428753 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:36.428786 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:43:36.428864 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:36.428871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:36.429091 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:36.429263 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:36.430858 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.432014 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.433289 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.434159 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.434354 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:36.438935 (+ 4581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:36.438949 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:36.438972 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:36.439011 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:36.442062 (+ 3051us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.442887 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.442896 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.442898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.443029 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.443539 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.443550 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.443552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.443898 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:36.444225 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:36.444584 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 02:43:36.444943 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":490,"thread_start_us":326,"threads_started":1}
W20260925 02:43:36.446818 5504 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42426
Sep 25 02:43:36 dist-test-slave-n563 krb5kdc[5353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:43:36 dist-test-slave-n563 krb5kdc[5353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:43:36.492499 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:36.476493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42436 (local address 127.5.56.129:35829)
0925 02:43:36.476703 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:36.476711 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:36.477060 (+ 349us) server_negotiation.cc:407] Connection header received
0925 02:43:36.477280 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:36.477285 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:36.477351 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:36.477463 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:36.478926 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.479688 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.480923 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.481354 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.481501 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:36.485894 (+ 4393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:36.485919 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:36.485927 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:36.485977 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:36.489036 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.489916 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.489923 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.489926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.490190 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.490712 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.490718 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.490721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.491054 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:36.491406 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:36.491733 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 02:43:36.492029 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:43:36.494033 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 5461
I20260925 02:43:36.508486 5868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:36.496088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42450 (local address 127.5.56.129:35829)
0925 02:43:36.496254 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:36.496258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:36.496577 (+ 319us) server_negotiation.cc:407] Connection header received
0925 02:43:36.496698 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:36.496702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:36.496756 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:36.496831 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:36.498469 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.499262 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.500572 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.501104 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.501198 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:36.503433 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:36.503438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:36.503441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:36.503470 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:36.506224 (+ 2754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.506899 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.506903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.506905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.507046 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.507353 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.507356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.507358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.507602 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:36.507774 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:36.508061 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 02:43:36.508214 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:43:36.509259 5585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:43:36.594363 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 5461
I20260925 02:43:36.611984 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 5596
Sep 25 02:43:36 dist-test-slave-n563 krb5kdc[5353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:43:36.627769 5875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:36.612916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55400 (local address 127.5.56.130:37087)
0925 02:43:36.613535 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:36.613542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:36.613569 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:43:36.613628 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:36.613635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:36.613847 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:36.613974 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:36.615346 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.616345 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.617568 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.618140 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.618238 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:36.622332 (+ 4094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:36.622345 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:36.622361 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:36.622451 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:36.625123 (+ 2672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.625762 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.625768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.625770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.625895 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.626253 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.626256 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.626258 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.626659 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:36.626883 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:36.627144 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 02:43:36.627351 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":334,"threads_started":1}
I20260925 02:43:36.629128 5720 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:43:36.712421 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 5596
I20260925 02:43:36.729955 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 5731
Sep 25 02:43:36 dist-test-slave-n563 krb5kdc[5353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:43:36.747880 5877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:36.731012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.5.56.131:40061)
0925 02:43:36.731655 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:36.731662 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:36.731692 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:43:36.731755 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:36.731761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:36.731973 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:36.732169 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:36.733458 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.734734 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.736029 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:36.736615 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:36.736762 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:36.741326 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:36.741339 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:36.741354 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:36.741394 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:36.744363 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.745062 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.745070 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.745072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.745200 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:36.745840 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:36.745849 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:36.745853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:36.746283 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:36.746695 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:36.747181 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 02:43:36.747488 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":501,"thread_start_us":329,"threads_started":1}
I20260925 02:43:36.749266 5855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:43:36.832706 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 5731
I20260925 02:43:36.849552 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 5366
I20260925 02:43:36.850905 5429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:43:36.955788 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 5366
2026-09-25T02:43:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4141 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:43:37 dist-test-slave-n563 krb5kdc[5882](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:43:37 dist-test-slave-n563 krb5kdc[5882](info): set up 2 sockets
Sep 25 02:43:37 dist-test-slave-n563 krb5kdc[5882](info): commencing operation
krb5kdc: starting...
W20260925 02:43:38.039563 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:43:38 dist-test-slave-n563 krb5kdc[5882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:43:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:43:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:43:38.226848 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:42455
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:42455
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:38.547883 5895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:38.548195 5895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:38.548374 5895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:38.555588 5895 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:43:38.555699 5895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:38.555802 5895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:38.555914 5895 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:43:38.555994 5895 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:43:38.565943 5895 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:42455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:42455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:38.567404 5895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:38.568756 5895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:38.580704 5900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:38.580739 5901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:38.580654 5903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:38.581388 5895 server_base.cc:1034] running on GCE node
I20260925 02:43:38.582226 5895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:38.583712 5895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:38.584926 5895 hybrid_clock.cc:648] HybridClock initialized: now 1790304218584878 us; error 27 us; skew 500 ppm
I20260925 02:43:38.585294 5895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:38 dist-test-slave-n563 krb5kdc[5882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304218, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:38.594010 5895 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:43:38.595821 5895 webserver.cc:460] Webserver started at http://127.5.56.190:38727/ using document root <none> and password file <none>
I20260925 02:43:38.596580 5895 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:38.596743 5895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:38.597080 5895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:38.600230 5895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "34f8a94432ae47d1a9c1f354c5ecc52f"
format_stamp: "Formatted at 2026-09-25 02:43:38 on dist-test-slave-n563"
server_key: "1bdc4754844ef872e817736540294cd2"
server_key_iv: "c8dec29d6f9f971be62507b37ad12204"
server_key_version: "encryptionkey@0"
I20260925 02:43:38.600989 5895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "34f8a94432ae47d1a9c1f354c5ecc52f"
format_stamp: "Formatted at 2026-09-25 02:43:38 on dist-test-slave-n563"
server_key: "1bdc4754844ef872e817736540294cd2"
server_key_iv: "c8dec29d6f9f971be62507b37ad12204"
server_key_version: "encryptionkey@0"
I20260925 02:43:38.606289 5895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 02:43:38.609835 5910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:38.611267 5895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:43:38.611464 5895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "34f8a94432ae47d1a9c1f354c5ecc52f"
format_stamp: "Formatted at 2026-09-25 02:43:38 on dist-test-slave-n563"
server_key: "1bdc4754844ef872e817736540294cd2"
server_key_iv: "c8dec29d6f9f971be62507b37ad12204"
server_key_version: "encryptionkey@0"
I20260925 02:43:38.611688 5895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:38.630146 5895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:38.634284 5895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:38.634620 5895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:38.653769 5895 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:42455
I20260925 02:43:38.653807 5962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:42455 every 8 connection(s)
I20260925 02:43:38.655296 5895 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:43:38.659668 5963 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:38.664661 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 5895
I20260925 02:43:38.664942 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:43:38.665645 5346 external_mini_cluster.cc:1428] Setting key 31f66d7eae64d258c23d594f6a0366f8
I20260925 02:43:38.671007 5963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f: Bootstrap starting.
I20260925 02:43:38.674983 5963 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:38.676262 5963 log.cc:824] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:38.679927 5963 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f: No bootstrap required, opened a new log
Sep 25 02:43:38 dist-test-slave-n563 krb5kdc[5882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:38.688001 5963 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f8a94432ae47d1a9c1f354c5ecc52f" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42455 } }
I20260925 02:43:38.688402 5963 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:38.688542 5963 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f8a94432ae47d1a9c1f354c5ecc52f, State: Initialized, Role: FOLLOWER
I20260925 02:43:38.689227 5963 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f8a94432ae47d1a9c1f354c5ecc52f" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42455 } }
I20260925 02:43:38.689496 5963 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:43:38.689670 5963 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:43:38.689872 5963 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:38.690274 5966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:38.667809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38682 (local address 127.5.56.190:42455)
0925 02:43:38.668762 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:38.668777 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:38.668821 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:43:38.670636 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:38.670660 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:38.671071 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:38.671467 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:38.673389 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:38.674749 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:38.676560 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:38.677700 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:38.677841 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:38.682539 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:38.682548 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:38.682564 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:38.682611 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:38.685795 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:38.686537 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:38.686544 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:38.686551 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:38.686731 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:38.687132 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:38.687144 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:38.687146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:38.687698 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:38.688005 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:38.688504 (+ 499us) server_negotiation.cc:299] Negotiation successful
0925 02:43:38.689018 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":445,"threads_started":1}
I20260925 02:43:38.692968 5963 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f8a94432ae47d1a9c1f354c5ecc52f" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42455 } }
I20260925 02:43:38.693590 5963 leader_election.cc:302] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34f8a94432ae47d1a9c1f354c5ecc52f; no voters:
I20260925 02:43:38.694206 5963 leader_election.cc:288] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:43:38.694342 5968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:38.695130 5968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [term 1 LEADER]: Becoming Leader. State: Replica: 34f8a94432ae47d1a9c1f354c5ecc52f, State: Running, Role: LEADER
I20260925 02:43:38.695670 5968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f8a94432ae47d1a9c1f354c5ecc52f" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42455 } }
I20260925 02:43:38.696388 5963 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:43:38.699173 5969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34f8a94432ae47d1a9c1f354c5ecc52f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f8a94432ae47d1a9c1f354c5ecc52f" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42455 } } }
I20260925 02:43:38.699153 5970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34f8a94432ae47d1a9c1f354c5ecc52f. Latest consensus state: current_term: 1 leader_uuid: "34f8a94432ae47d1a9c1f354c5ecc52f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f8a94432ae47d1a9c1f354c5ecc52f" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42455 } } }
I20260925 02:43:38.699690 5970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [sys.catalog]: This master's current role is: LEADER
I20260925 02:43:38.699671 5969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f [sys.catalog]: This master's current role is: LEADER
I20260925 02:43:38.701105 5973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:43:38.707590 5973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:43:38.717067 5973 catalog_manager.cc:1347] Generated new cluster ID: f7c57ee9deb743e588dbc465e4200b5a
I20260925 02:43:38.717180 5973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:43:38.723803 5973 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:43:38.725551 5973 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:43:38.736243 5973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34f8a94432ae47d1a9c1f354c5ecc52f: Generated new TSK 0
I20260925 02:43:38.737304 5973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:43:38.785434 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42455
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:39.118440 5989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:39.118731 5989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:39.118827 5989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:39.126159 5989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:39.126281 5989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:39.126416 5989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:43:39.136349 5989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42455
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:39.137622 5989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:39.139029 5989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:39.152685 5995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:39.152732 5997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:39.152719 5994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:39.153441 5989 server_base.cc:1034] running on GCE node
I20260925 02:43:39.154000 5989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:39.155004 5989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:39.156214 5989 hybrid_clock.cc:648] HybridClock initialized: now 1790304219156172 us; error 28 us; skew 500 ppm
I20260925 02:43:39.156539 5989 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:39 dist-test-slave-n563 krb5kdc[5882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:39.159757 5989 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:43:39.161234 5989 webserver.cc:460] Webserver started at http://127.5.56.129:38711/ using document root <none> and password file <none>
I20260925 02:43:39.161929 5989 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:39.162078 5989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:39.162463 5989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:39.165397 5989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "e6fae55fcb6d4726a26edbb2cd7cff97"
format_stamp: "Formatted at 2026-09-25 02:43:39 on dist-test-slave-n563"
server_key: "de268a07c26bdfeda50f90815e2ce23d"
server_key_iv: "716a4b3446f11de39957cd414144a112"
server_key_version: "encryptionkey@0"
I20260925 02:43:39.166191 5989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "e6fae55fcb6d4726a26edbb2cd7cff97"
format_stamp: "Formatted at 2026-09-25 02:43:39 on dist-test-slave-n563"
server_key: "de268a07c26bdfeda50f90815e2ce23d"
server_key_iv: "716a4b3446f11de39957cd414144a112"
server_key_version: "encryptionkey@0"
I20260925 02:43:39.171486 5989 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 02:43:39.175091 6004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:39.176582 5989 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:43:39.176764 5989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "e6fae55fcb6d4726a26edbb2cd7cff97"
format_stamp: "Formatted at 2026-09-25 02:43:39 on dist-test-slave-n563"
server_key: "de268a07c26bdfeda50f90815e2ce23d"
server_key_iv: "716a4b3446f11de39957cd414144a112"
server_key_version: "encryptionkey@0"
I20260925 02:43:39.176972 5989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:39.199807 5989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:39.204487 5989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:39.204792 5989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:39.205881 5989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:43:39.207393 5989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:39.207500 5989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:39.207646 5989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:39.207719 5989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:39.250233 5989 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:40481
I20260925 02:43:39.250348 6116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:40481 every 8 connection(s)
I20260925 02:43:39.251621 5989 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:43:39.254279 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 5989
I20260925 02:43:39.254551 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:43:39.255379 5346 external_mini_cluster.cc:1428] Setting key f40ca02de841f5c78f25baab7406c817
Sep 25 02:43:39 dist-test-slave-n563 krb5kdc[5882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
I20260925 02:43:39.275678 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:39.254949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:51075 (local address 127.5.56.190:42455)
0925 02:43:39.255456 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:39.255464 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:39.258466 (+ 3002us) server_negotiation.cc:407] Connection header received
0925 02:43:39.259563 (+ 1097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:39.259573 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:39.259678 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:39.259803 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:39.261727 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:39.262948 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:39.265203 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:39.265650 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:39.265802 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:39.269551 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:39.269556 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:39.269560 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:39.269597 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:39.272369 (+ 2772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:39.273160 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:39.273166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:39.273170 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:39.273334 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:39.273779 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:39.273786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:39.273789 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:39.274071 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:39.274228 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:39.275047 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 02:43:39.275287 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":224,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
I20260925 02:43:39.276778 6120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:39.255652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42455 (local address 127.5.56.129:51075)
0925 02:43:39.256635 (+ 983us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:39.256690 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:39.259215 (+ 2525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:39.260138 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:39.260151 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:39.260765 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:39.261489 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:39.261519 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:39.263161 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:39.263170 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:39.265029 (+ 1859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:39.265045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:39.265925 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:39.265935 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:39.266199 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:39.267006 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:39.267031 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:39.269280 (+ 2249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:39.272538 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:39.272548 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:39.272567 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:39.273032 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:39.273461 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:39.273466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:39.273469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:39.273665 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:39.274202 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:39.274211 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:39.274657 (+ 446us) client_negotiation.cc:769] Sending connection context
0925 02:43:39.274940 (+ 283us) client_negotiation.cc:240] Negotiation successful
0925 02:43:39.275548 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":389,"threads_started":1}
I20260925 02:43:39.278460 6117 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42455
I20260925 02:43:39.278812 6117 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:39.279695 6117 heartbeater.cc:511] Master 127.5.56.190:42455 requested a full tablet report, sending...
I20260925 02:43:39.282219 5927 ts_manager.cc:194] Registered new tserver with Master: e6fae55fcb6d4726a26edbb2cd7cff97 (127.5.56.129:40481)
I20260925 02:43:39.284466 5927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:51075
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:43:39.302119 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42455
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:39.624469 6123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:39.624764 6123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:39.624862 6123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:39.632124 6123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:39.632242 6123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:39.632357 6123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:43:39.642925 6123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42455
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:39.645010 6123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:39.646994 6123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:39.660566 6129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:39.661161 6131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:39.661265 6128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:39.661659 6123 server_base.cc:1034] running on GCE node
I20260925 02:43:39.662348 6123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:39.663440 6123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:39.664687 6123 hybrid_clock.cc:648] HybridClock initialized: now 1790304219664630 us; error 43 us; skew 500 ppm
I20260925 02:43:39.665052 6123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:39 dist-test-slave-n563 krb5kdc[5882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:39.668810 6123 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:43:39.672783 6123 webserver.cc:460] Webserver started at http://127.5.56.130:38803/ using document root <none> and password file <none>
I20260925 02:43:39.673506 6123 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:39.673653 6123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:39.673974 6123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:39.677026 6123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2"
format_stamp: "Formatted at 2026-09-25 02:43:39 on dist-test-slave-n563"
server_key: "28f7594164759a30b5d05181c9e3ee4b"
server_key_iv: "87fa41f4f3ca2be4b4bf258498451af0"
server_key_version: "encryptionkey@0"
I20260925 02:43:39.677811 6123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2"
format_stamp: "Formatted at 2026-09-25 02:43:39 on dist-test-slave-n563"
server_key: "28f7594164759a30b5d05181c9e3ee4b"
server_key_iv: "87fa41f4f3ca2be4b4bf258498451af0"
server_key_version: "encryptionkey@0"
I20260925 02:43:39.683451 6123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 02:43:39.686852 6138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:39.688184 6123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:43:39.688359 6123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2"
format_stamp: "Formatted at 2026-09-25 02:43:39 on dist-test-slave-n563"
server_key: "28f7594164759a30b5d05181c9e3ee4b"
server_key_iv: "87fa41f4f3ca2be4b4bf258498451af0"
server_key_version: "encryptionkey@0"
I20260925 02:43:39.688567 6123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:39.716571 6123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:39.721544 6123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:39.721855 6123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:39.722906 6123 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:43:39.724303 6123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:39.724426 6123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:39.724581 6123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:39.724654 6123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:39.757598 6123 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:39197
I20260925 02:43:39.757647 6250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:39197 every 8 connection(s)
I20260925 02:43:39.758998 6123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:43:39.760360 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 6123
I20260925 02:43:39.760599 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:43:39.761204 5346 external_mini_cluster.cc:1428] Setting key 02dd736b4e5fb01a9ffa7babe3c9c461
Sep 25 02:43:39 dist-test-slave-n563 krb5kdc[5882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304219, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
I20260925 02:43:39.782984 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:39.762184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:35501 (local address 127.5.56.190:42455)
0925 02:43:39.762439 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:39.762446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:39.763803 (+ 1357us) server_negotiation.cc:407] Connection header received
0925 02:43:39.765802 (+ 1999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:39.765811 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:39.765914 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:39.766030 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:39.767711 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:39.768557 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:39.771854 (+ 3297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:39.772278 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:39.772372 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:39.776917 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:39.776923 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:39.776927 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:39.776961 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:39.779712 (+ 2751us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:39.780493 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:39.780498 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:39.780501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:39.780670 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:39.781111 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:39.781116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:39.781119 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:39.781401 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:39.781553 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:39.782356 (+ 803us) server_negotiation.cc:299] Negotiation successful
0925 02:43:39.782632 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:43:39.784006 6253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:39.762701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42455 (local address 127.5.56.130:35501)
0925 02:43:39.763623 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:39.763680 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:39.765502 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:39.766261 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:39.766270 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:39.766907 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:39.767527 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:39.767547 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:39.768741 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:39.768751 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:39.771700 (+ 2949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:39.771720 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:39.772544 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:39.772554 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:39.772914 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:39.773712 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:39.773751 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:39.776664 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:39.779859 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:39.779869 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:39.779888 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:39.780354 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:39.780784 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:39.780789 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:39.780793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:39.780980 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:39.781539 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:39.781547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:39.781992 (+ 445us) client_negotiation.cc:769] Sending connection context
0925 02:43:39.782262 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 02:43:39.782841 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":333,"threads_started":1}
I20260925 02:43:39.785625 6251 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42455
I20260925 02:43:39.785936 6251 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:39.786706 6251 heartbeater.cc:511] Master 127.5.56.190:42455 requested a full tablet report, sending...
I20260925 02:43:39.788120 5927 ts_manager.cc:194] Registered new tserver with Master: 719f68d06b8d49a5a7ba1b2f6fde38e2 (127.5.56.130:39197)
I20260925 02:43:39.788972 5927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:35501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:43:39.805868 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42455
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:40.140409 6256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:40.140704 6256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:40.140802 6256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:40.148131 6256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:40.148255 6256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:40.148376 6256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:43:40.158674 6256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42455
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:40.159943 6256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:40.161285 6256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:40.174741 6261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:40.174813 6262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:40.174853 6264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:40.175352 6256 server_base.cc:1034] running on GCE node
I20260925 02:43:40.176033 6256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:40.176959 6256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:40.178205 6256 hybrid_clock.cc:648] HybridClock initialized: now 1790304220178169 us; error 47 us; skew 500 ppm
I20260925 02:43:40.178633 6256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:40 dist-test-slave-n563 krb5kdc[5882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:40.181945 6256 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:43:40.183480 6256 webserver.cc:460] Webserver started at http://127.5.56.131:38309/ using document root <none> and password file <none>
I20260925 02:43:40.184211 6256 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:40.184342 6256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:40.184687 6256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:40.187881 6256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "3b96e1b0805e4d5eb652248b45c0be47"
format_stamp: "Formatted at 2026-09-25 02:43:40 on dist-test-slave-n563"
server_key: "55392dd874d2d41a034c1527b8a14059"
server_key_iv: "6067f032d241b6d6fc3c4f7e1fe7a921"
server_key_version: "encryptionkey@0"
I20260925 02:43:40.188683 6256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "3b96e1b0805e4d5eb652248b45c0be47"
format_stamp: "Formatted at 2026-09-25 02:43:40 on dist-test-slave-n563"
server_key: "55392dd874d2d41a034c1527b8a14059"
server_key_iv: "6067f032d241b6d6fc3c4f7e1fe7a921"
server_key_version: "encryptionkey@0"
I20260925 02:43:40.194080 6256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:43:40.198869 6271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:40.200253 6256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:43:40.200446 6256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "3b96e1b0805e4d5eb652248b45c0be47"
format_stamp: "Formatted at 2026-09-25 02:43:40 on dist-test-slave-n563"
server_key: "55392dd874d2d41a034c1527b8a14059"
server_key_iv: "6067f032d241b6d6fc3c4f7e1fe7a921"
server_key_version: "encryptionkey@0"
I20260925 02:43:40.200659 6256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:40.214958 6256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:40.219584 6256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:40.219887 6256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:40.220857 6256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:43:40.222179 6256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:40.222283 6256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:40.222456 6256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:40.222546 6256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 02:43:40.256862 6256 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:42627
I20260925 02:43:40.256881 6383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:42627 every 8 connection(s)
I20260925 02:43:40.258322 6256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:43:40.264145 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 6256
I20260925 02:43:40.264359 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:43:40.264781 5346 external_mini_cluster.cc:1428] Setting key 7f1307f25ef8fe3029663f0d928b6a73
Sep 25 02:43:40 dist-test-slave-n563 krb5kdc[5882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304220, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:40.279393 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.261126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:38655 (local address 127.5.56.190:42455)
0925 02:43:40.261330 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:40.261335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:40.262768 (+ 1433us) server_negotiation.cc:407] Connection header received
0925 02:43:40.264821 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:40.264827 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:40.264947 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:40.265063 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:40.267256 (+ 2193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.268121 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.269687 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.270091 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.270231 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.273701 (+ 3470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:40.273707 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:40.273711 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:40.273743 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:40.276225 (+ 2482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:40.276962 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:40.276966 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:40.276969 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:40.277096 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:40.277547 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:40.277551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:40.277553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:40.277832 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:40.277993 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:40.278843 (+ 850us) server_negotiation.cc:299] Negotiation successful
0925 02:43:40.279128 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:43:40.280680 6386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.261597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42455 (local address 127.5.56.131:38655)
0925 02:43:40.262573 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:40.262632 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:40.264532 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:40.265333 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:40.265342 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:40.266306 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:40.267085 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.267105 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.268275 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.268287 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.269522 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.269534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.270335 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.270342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.270611 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.271243 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:40.271266 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:40.273438 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:40.276370 (+ 2932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:40.276380 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:40.276397 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:40.276821 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:40.277217 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:40.277222 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:40.277226 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:40.277416 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:40.277967 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:40.277976 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:40.278322 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 02:43:40.278685 (+ 363us) client_negotiation.cc:240] Negotiation successful
0925 02:43:40.279224 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":463,"threads_started":1}
I20260925 02:43:40.282043 6384 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42455
I20260925 02:43:40.282357 6384 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:40.283131 6384 heartbeater.cc:511] Master 127.5.56.190:42455 requested a full tablet report, sending...
I20260925 02:43:40.284493 5927 ts_manager.cc:194] Registered new tserver with Master: 3b96e1b0805e4d5eb652248b45c0be47 (127.5.56.131:42627)
I20260925 02:43:40.285281 5927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:38655
I20260925 02:43:40.287756 6117 heartbeater.cc:503] Master 127.5.56.190:42455 was elected leader, sending a full tablet report...
I20260925 02:43:40.292369 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:43:40.317886 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.305791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38694 (local address 127.5.56.190:42455)
0925 02:43:40.306039 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:40.306049 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:40.306438 (+ 389us) server_negotiation.cc:407] Connection header received
0925 02:43:40.306637 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:40.306641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:40.306710 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:40.306796 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:40.308286 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.309012 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.310217 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.310678 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.310771 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.313045 (+ 2274us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:40.313050 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:40.313054 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:40.313086 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:40.315607 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:40.316239 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:40.316243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:40.316245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:40.316374 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:40.316669 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:40.316672 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:40.316674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:40.316962 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:40.317153 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:40.317390 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 02:43:40.317546 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:43:40.329229 5927 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:43:40.336655 5927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:43:40.361295 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.352673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46558 (local address 127.5.56.129:40481)
0925 02:43:40.353785 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:40.353792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:40.353816 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:43:40.353906 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:40.353937 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:40.354178 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:40.354429 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:40.354986 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.356448 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.358708 (+ 2260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.360112 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.360272 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.360525 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:40.360774 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 02:43:40.360934 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":950,"thread_start_us":1259,"threads_started":1}
I20260925 02:43:40.361443 6396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.350375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39197 (local address 127.0.0.1:49246)
0925 02:43:40.351190 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:40.351238 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:40.351455 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:40.353305 (+ 1850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:40.353315 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:40.353355 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:40.353791 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.353807 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.355786 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.355791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.358537 (+ 2746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.358552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.360495 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.360503 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.360693 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.360743 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 02:43:40.360801 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 02:43:40.360928 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"mutex_wait_us":127,"thread_start_us":646,"threads_started":1}
I20260925 02:43:40.361963 6400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.352026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:40481 (local address 127.0.0.1:46558)
0925 02:43:40.353121 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:40.353148 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:40.353260 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:40.354426 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:40.354431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:40.354458 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:40.354807 (+ 349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.354819 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.356622 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.356631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.358553 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.358566 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.360301 (+ 1735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.360307 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.360510 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.360573 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 02:43:40.360732 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 02:43:40.360934 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"mutex_wait_us":61,"thread_start_us":847,"threads_started":1}
I20260925 02:43:40.362046 6398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.350847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:42627 (local address 127.0.0.1:40224)
0925 02:43:40.352450 (+ 1603us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:40.352469 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:40.352594 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:40.353542 (+ 948us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:40.353546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:40.353567 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:40.354025 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.354035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.356047 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.356053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.359027 (+ 2974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.359041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.361147 (+ 2106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.361153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.361345 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.361373 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:43:40.361471 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 02:43:40.361587 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1481,"mutex_wait_us":203,"thread_start_us":1711,"threads_started":1}
I20260925 02:43:40.362425 6397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.350530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.5.56.130:39197)
0925 02:43:40.352669 (+ 2139us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:40.352678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:40.352707 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:43:40.352807 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:40.352814 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:40.353062 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:40.354042 (+ 980us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:40.354279 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.355629 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.358750 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.360251 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.361124 (+ 873us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.361312 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:40.361866 (+ 554us) server_negotiation.cc:299] Negotiation successful
0925 02:43:40.362071 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1961,"thread_start_us":1832,"threads_started":1}
I20260925 02:43:40.362958 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.351118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40224 (local address 127.5.56.131:42627)
0925 02:43:40.352197 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:40.352210 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:40.352728 (+ 518us) server_negotiation.cc:407] Connection header received
0925 02:43:40.352819 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:40.352825 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:40.353061 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:40.353256 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:40.354198 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.355877 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.359229 (+ 3352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.360651 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.360771 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.360989 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:40.362331 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0925 02:43:40.362650 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":744,"threads_started":1}
I20260925 02:43:40.365818 6052 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1b5ec95ecb4b79af1de81ae34b2da4 (DEFAULT_TABLE table=test-table [id=87b281de1d2141fdbefd2d7d95d695c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:40.366245 6319 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1b5ec95ecb4b79af1de81ae34b2da4 (DEFAULT_TABLE table=test-table [id=87b281de1d2141fdbefd2d7d95d695c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:40.367436 6052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1b5ec95ecb4b79af1de81ae34b2da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:40.367532 6319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1b5ec95ecb4b79af1de81ae34b2da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:40.367830 6186 tablet_service.cc:1467] Processing CreateTablet for tablet 5f1b5ec95ecb4b79af1de81ae34b2da4 (DEFAULT_TABLE table=test-table [id=87b281de1d2141fdbefd2d7d95d695c0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:40.369113 6186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f1b5ec95ecb4b79af1de81ae34b2da4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:40.379314 6402 tablet_bootstrap.cc:492] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47: Bootstrap starting.
I20260925 02:43:40.382692 6403 tablet_bootstrap.cc:492] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97: Bootstrap starting.
I20260925 02:43:40.384227 6404 tablet_bootstrap.cc:492] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2: Bootstrap starting.
I20260925 02:43:40.384667 6402 tablet_bootstrap.cc:654] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:40.386067 6402 log.cc:824] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:40.386286 6403 tablet_bootstrap.cc:654] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:40.387279 6404 tablet_bootstrap.cc:654] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:40.387645 6403 log.cc:824] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:40.388491 6404 log.cc:824] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:40.389752 6402 tablet_bootstrap.cc:492] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47: No bootstrap required, opened a new log
I20260925 02:43:40.390105 6402 ts_tablet_manager.cc:1397] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 02:43:40.391155 6403 tablet_bootstrap.cc:492] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97: No bootstrap required, opened a new log
I20260925 02:43:40.391250 6404 tablet_bootstrap.cc:492] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2: No bootstrap required, opened a new log
I20260925 02:43:40.391440 6403 ts_tablet_manager.cc:1397] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260925 02:43:40.391491 6404 ts_tablet_manager.cc:1397] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20260925 02:43:40.399765 6404 raft_consensus.cc:348] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39197 } } peers { permanent_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 40481 } } peers { permanent_uuid: "3b96e1b0805e4d5eb652248b45c0be47" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 42627 } }
I20260925 02:43:40.400185 6404 raft_consensus.cc:374] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:40.400280 6404 raft_consensus.cc:729] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719f68d06b8d49a5a7ba1b2f6fde38e2, State: Initialized, Role: FOLLOWER
I20260925 02:43:40.400889 6404 consensus_queue.cc:260] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39197 } } peers { permanent_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 40481 } } peers { permanent_uuid: "3b96e1b0805e4d5eb652248b45c0be47" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 42627 } }
I20260925 02:43:40.401257 6402 raft_consensus.cc:348] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39197 } } peers { permanent_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 40481 } } peers { permanent_uuid: "3b96e1b0805e4d5eb652248b45c0be47" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 42627 } }
I20260925 02:43:40.401749 6402 raft_consensus.cc:374] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:40.401868 6402 raft_consensus.cc:729] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b96e1b0805e4d5eb652248b45c0be47, State: Initialized, Role: FOLLOWER
I20260925 02:43:40.402104 6403 raft_consensus.cc:348] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39197 } } peers { permanent_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 40481 } } peers { permanent_uuid: "3b96e1b0805e4d5eb652248b45c0be47" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 42627 } }
I20260925 02:43:40.402513 6251 heartbeater.cc:503] Master 127.5.56.190:42455 was elected leader, sending a full tablet report...
I20260925 02:43:40.402578 6403 raft_consensus.cc:374] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:40.402680 6403 raft_consensus.cc:729] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6fae55fcb6d4726a26edbb2cd7cff97, State: Initialized, Role: FOLLOWER
I20260925 02:43:40.402626 6402 consensus_queue.cc:260] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39197 } } peers { permanent_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 40481 } } peers { permanent_uuid: "3b96e1b0805e4d5eb652248b45c0be47" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 42627 } }
I20260925 02:43:40.403225 6404 ts_tablet_manager.cc:1428] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2: Time spent starting tablet: real 0.012s user 0.010s sys 0.003s
I20260925 02:43:40.403255 6403 consensus_queue.cc:260] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39197 } } peers { permanent_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 40481 } } peers { permanent_uuid: "3b96e1b0805e4d5eb652248b45c0be47" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 42627 } }
I20260925 02:43:40.404486 6384 heartbeater.cc:503] Master 127.5.56.190:42455 was elected leader, sending a full tablet report...
I20260925 02:43:40.405632 6402 ts_tablet_manager.cc:1428] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260925 02:43:40.407848 6403 ts_tablet_manager.cc:1428] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97: Time spent starting tablet: real 0.016s user 0.007s sys 0.009s
W20260925 02:43:40.504627 6118 tablet.cc:2369] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:43:40.510000 6385 tablet.cc:2369] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:43:40.511421 6252 tablet.cc:2369] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:43:40.797799 6408 raft_consensus.cc:482] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:43:40.798173 6408 raft_consensus.cc:504] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39197 } } peers { permanent_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 40481 } } peers { permanent_uuid: "3b96e1b0805e4d5eb652248b45c0be47" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 42627 } }
I20260925 02:43:40.800256 6408 leader_election.cc:288] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e6fae55fcb6d4726a26edbb2cd7cff97 (127.5.56.129:40481), 3b96e1b0805e4d5eb652248b45c0be47 (127.5.56.131:42627)
I20260925 02:43:40.807675 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.800619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:46019 (local address 127.5.56.129:40481)
0925 02:43:40.800846 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:40.800853 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:40.801824 (+ 971us) server_negotiation.cc:407] Connection header received
0925 02:43:40.802003 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:40.802007 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:40.802088 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:40.802194 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:40.802960 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.803742 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.805810 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.806679 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.806839 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.806934 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:40.807285 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 02:43:40.807410 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:43:40.807704 6399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.800671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:48671 (local address 127.5.56.131:42627)
0925 02:43:40.800891 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:40.800899 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:40.801640 (+ 741us) server_negotiation.cc:407] Connection header received
0925 02:43:40.801814 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:40.801820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:40.801900 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:40.801994 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:40.802694 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.803479 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.805858 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.806697 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.806881 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.806952 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:40.807321 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 02:43:40.807450 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:43:40.807806 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.800705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:42627 (local address 127.5.56.130:48671)
0925 02:43:40.801516 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:40.801533 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:40.801667 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:40.802029 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:40.802033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:40.802056 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:40.802513 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.802524 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.803630 (+ 1106us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.803636 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.805682 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.805694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.806962 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.806969 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.807143 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.807166 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:43:40.807274 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:43:40.807401 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":48,"thread_start_us":435,"threads_started":1}
I20260925 02:43:40.808218 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.800723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:40481 (local address 127.5.56.130:46019)
0925 02:43:40.801683 (+ 960us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:40.801700 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:40.801852 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:40.802248 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:40.802252 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:40.802273 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:40.802697 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.802708 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.803882 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.803886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.805661 (+ 1775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:40.805673 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.806905 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.806911 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:40.807096 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.807150 (+ 54us) client_negotiation.cc:769] Sending connection context
0925 02:43:40.807225 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:43:40.807368 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"mutex_wait_us":75,"thread_start_us":560,"threads_started":1}
I20260925 02:43:40.808883 6339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1b5ec95ecb4b79af1de81ae34b2da4" candidate_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b96e1b0805e4d5eb652248b45c0be47" is_pre_election: true
I20260925 02:43:40.809327 6072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1b5ec95ecb4b79af1de81ae34b2da4" candidate_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97" is_pre_election: true
I20260925 02:43:40.809451 6339 raft_consensus.cc:2454] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 719f68d06b8d49a5a7ba1b2f6fde38e2 in term 0.
I20260925 02:43:40.809801 6072 raft_consensus.cc:2454] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 719f68d06b8d49a5a7ba1b2f6fde38e2 in term 0.
I20260925 02:43:40.810227 6139 leader_election.cc:302] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b96e1b0805e4d5eb652248b45c0be47, 719f68d06b8d49a5a7ba1b2f6fde38e2; no voters:
I20260925 02:43:40.810731 6408 raft_consensus.cc:2781] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:43:40.810897 6408 raft_consensus.cc:482] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:43:40.811033 6408 raft_consensus.cc:3037] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:40.813234 6408 raft_consensus.cc:504] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39197 } } peers { permanent_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 40481 } } peers { permanent_uuid: "3b96e1b0805e4d5eb652248b45c0be47" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 42627 } }
I20260925 02:43:40.813858 6408 leader_election.cc:288] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [CANDIDATE]: Term 1 election: Requested vote from peers e6fae55fcb6d4726a26edbb2cd7cff97 (127.5.56.129:40481), 3b96e1b0805e4d5eb652248b45c0be47 (127.5.56.131:42627)
I20260925 02:43:40.814291 6072 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1b5ec95ecb4b79af1de81ae34b2da4" candidate_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97"
I20260925 02:43:40.814339 6339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f1b5ec95ecb4b79af1de81ae34b2da4" candidate_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b96e1b0805e4d5eb652248b45c0be47"
I20260925 02:43:40.814548 6072 raft_consensus.cc:3037] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:40.814590 6339 raft_consensus.cc:3037] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:40.816938 6072 raft_consensus.cc:2454] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P e6fae55fcb6d4726a26edbb2cd7cff97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 719f68d06b8d49a5a7ba1b2f6fde38e2 in term 1.
I20260925 02:43:40.817059 6339 raft_consensus.cc:2454] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 3b96e1b0805e4d5eb652248b45c0be47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 719f68d06b8d49a5a7ba1b2f6fde38e2 in term 1.
I20260925 02:43:40.817384 6140 leader_election.cc:302] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 719f68d06b8d49a5a7ba1b2f6fde38e2, e6fae55fcb6d4726a26edbb2cd7cff97; no voters:
I20260925 02:43:40.817746 6408 raft_consensus.cc:2781] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:40.818279 6408 raft_consensus.cc:686] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [term 1 LEADER]: Becoming Leader. State: Replica: 719f68d06b8d49a5a7ba1b2f6fde38e2, State: Running, Role: LEADER
I20260925 02:43:40.818817 6408 consensus_queue.cc:237] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39197 } } peers { permanent_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 40481 } } peers { permanent_uuid: "3b96e1b0805e4d5eb652248b45c0be47" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 42627 } }
I20260925 02:43:40.823597 5925 catalog_manager.cc:5495] T 5f1b5ec95ecb4b79af1de81ae34b2da4 P 719f68d06b8d49a5a7ba1b2f6fde38e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 719f68d06b8d49a5a7ba1b2f6fde38e2 (127.5.56.130). New cstate: current_term: 1 leader_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719f68d06b8d49a5a7ba1b2f6fde38e2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39197 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6fae55fcb6d4726a26edbb2cd7cff97" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 40481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b96e1b0805e4d5eb652248b45c0be47" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 42627 } health_report { overall_health: UNKNOWN } } }
Sep 25 02:43:40 dist-test-slave-n563 krb5kdc[5882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304218, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:43:40.938472 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.924206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46568 (local address 127.5.56.129:40481)
0925 02:43:40.924567 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:40.924573 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:40.924795 (+ 222us) server_negotiation.cc:407] Connection header received
0925 02:43:40.924938 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:40.924942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:40.925008 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:40.925094 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:40.926476 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.927189 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.928437 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.928822 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.928912 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.932773 (+ 3861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:40.932788 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:40.932805 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:40.932842 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:40.935623 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:40.936320 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:40.936327 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:40.936329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:40.936459 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:40.936847 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:40.936852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:40.936855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:40.937212 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:40.937545 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:40.937834 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 02:43:40.938055 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
Sep 25 02:43:40 dist-test-slave-n563 krb5kdc[5882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:43:40 dist-test-slave-n563 krb5kdc[5882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304220, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:43:40.986660 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:40.972935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46578 (local address 127.5.56.129:40481)
0925 02:43:40.973234 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:40.973240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:40.973507 (+ 267us) server_negotiation.cc:407] Connection header received
0925 02:43:40.973660 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:40.973664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:40.973732 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:40.973822 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:40.975383 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.976099 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.977298 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:40.977683 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:40.977772 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:40.981747 (+ 3975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:40.981752 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:40.981756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:40.981791 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:40.984361 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:40.984943 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:40.984946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:40.984949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:40.985104 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:40.985442 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:40.985445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:40.985447 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:40.985704 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:40.985888 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:40.986149 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 02:43:40.986307 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
Sep 25 02:43:41 dist-test-slave-n563 krb5kdc[5882](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:43:41 dist-test-slave-n563 krb5kdc[5882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:43:41.034152 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:41.020619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46584 (local address 127.5.56.129:40481)
0925 02:43:41.020869 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:41.020876 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:41.021182 (+ 306us) server_negotiation.cc:407] Connection header received
0925 02:43:41.021415 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:41.021420 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:41.021485 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:41.021597 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:41.023008 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:41.023768 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:41.024979 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:41.025425 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:41.025565 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:41.029363 (+ 3798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:41.029368 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:41.029372 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:41.029407 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:41.031893 (+ 2486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:41.032551 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:41.032554 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:41.032556 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:41.032676 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:41.032975 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:41.032978 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:41.032980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:41.033267 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:41.033447 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:41.033692 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 02:43:41.033866 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 02:43:41.036182 6032 tablet_service.cc:2908] Rejecting scan request for tablet 5f1b5ec95ecb4b79af1de81ae34b2da4: Uninitialized: clean time has not yet been initialized
I20260925 02:43:41.039686 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 5989
I20260925 02:43:41.052291 6401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:41.040693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.5.56.129:40481)
0925 02:43:41.040932 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:41.040943 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:41.041345 (+ 402us) server_negotiation.cc:407] Connection header received
0925 02:43:41.041479 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:41.041484 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:41.041555 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:41.041654 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:41.042921 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:41.043632 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:41.044767 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:41.045175 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:41.045270 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:41.047581 (+ 2311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:41.047585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:41.047588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:41.047625 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:41.049994 (+ 2369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:41.050670 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:41.050674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:41.050676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:41.050788 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:41.051098 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:41.051101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:41.051103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:41.051364 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:41.051568 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:41.051765 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 02:43:41.051926 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 02:43:41.053005 6112 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:43:41.142897 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 5989
I20260925 02:43:41.160874 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 6123
Sep 25 02:43:41 dist-test-slave-n563 krb5kdc[5882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:43:41.177124 6427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:41.161680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.5.56.130:39197)
0925 02:43:41.162165 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:41.162170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:41.162186 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:43:41.162241 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:41.162245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:41.162304 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:41.162455 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:41.163630 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:41.164321 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:41.165772 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:41.166244 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:41.166365 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:41.170734 (+ 4369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:41.170752 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:41.170770 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:41.170811 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:41.173802 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:41.174729 (+ 927us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:41.174738 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:41.174740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:41.174898 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:41.175354 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:41.175358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:41.175360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:41.175756 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:41.176096 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:41.176455 (+ 359us) server_negotiation.cc:299] Negotiation successful
0925 02:43:41.176706 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":242,"threads_started":1}
I20260925 02:43:41.177853 6246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:43:41.289820 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 6123
I20260925 02:43:41.309593 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 6256
Sep 25 02:43:41 dist-test-slave-n563 krb5kdc[5882](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304221, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:43:41.325496 6429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:41.310435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40228 (local address 127.5.56.131:42627)
0925 02:43:41.310963 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:41.310968 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:41.310986 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:43:41.311041 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:41.311045 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:41.311121 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:41.311214 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:41.312693 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:41.313466 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:41.314846 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:41.315278 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:41.315375 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:41.319721 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:41.319737 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:41.319755 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:41.319794 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:41.322922 (+ 3128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:41.323635 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:41.323642 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:41.323644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:41.323773 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:41.324099 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:41.324109 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:41.324110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:41.324457 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:41.324683 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:41.324911 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 02:43:41.325087 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430,"thread_start_us":301,"threads_started":1}
I20260925 02:43:41.326207 6379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:43:41.415060 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 6256
W20260925 02:43:41.432550 5911 connection.cc:535] server connection from 127.5.56.131:38655 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:43:41.432760 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 5895
I20260925 02:43:41.433954 5958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:43:41.543481 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 5895
2026-09-25T02:43:41Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4587 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:43:41 dist-test-slave-n563 krb5kdc[6434](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:43:41 dist-test-slave-n563 krb5kdc[6434](info): set up 2 sockets
Sep 25 02:43:41 dist-test-slave-n563 krb5kdc[6434](info): commencing operation
krb5kdc: starting...
W20260925 02:43:42.631706 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:43:42 dist-test-slave-n563 krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:43:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:43:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:43:42.826679 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:46463
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:46463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:43.155980 6447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:43.156276 6447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:43.156369 6447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:43.163520 6447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:43:43.163645 6447 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:43:43.163686 6447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:43.163750 6447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:43.163803 6447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:43:43.163865 6447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:43:43.173851 6447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:46463
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:46463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:43.175158 6447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:43.176443 6447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:43.188769 6452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:43.188834 6453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:43.188812 6455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:43.189312 6447 server_base.cc:1034] running on GCE node
I20260925 02:43:43.190214 6447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:43.191810 6447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:43.193060 6447 hybrid_clock.cc:648] HybridClock initialized: now 1790304223193002 us; error 39 us; skew 500 ppm
I20260925 02:43:43.193418 6447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:43 dist-test-slave-n563 krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:43.196681 6447 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:43:43.198249 6447 webserver.cc:460] Webserver started at http://127.5.56.190:44311/ using document root <none> and password file <none>
I20260925 02:43:43.199043 6447 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:43.199186 6447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:43.199528 6447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:43.202667 6447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "30ba619501044e739ae9cbc5f46a6c68"
format_stamp: "Formatted at 2026-09-25 02:43:43 on dist-test-slave-n563"
server_key: "67f508f59f325de3e8c172ef22c3eb75"
server_key_iv: "207de3f7b4c1492ba65b406dbaf1c812"
server_key_version: "encryptionkey@0"
I20260925 02:43:43.203578 6447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "30ba619501044e739ae9cbc5f46a6c68"
format_stamp: "Formatted at 2026-09-25 02:43:43 on dist-test-slave-n563"
server_key: "67f508f59f325de3e8c172ef22c3eb75"
server_key_iv: "207de3f7b4c1492ba65b406dbaf1c812"
server_key_version: "encryptionkey@0"
I20260925 02:43:43.209971 6447 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260925 02:43:43.213559 6462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:43.214998 6447 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:43:43.215183 6447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "30ba619501044e739ae9cbc5f46a6c68"
format_stamp: "Formatted at 2026-09-25 02:43:43 on dist-test-slave-n563"
server_key: "67f508f59f325de3e8c172ef22c3eb75"
server_key_iv: "207de3f7b4c1492ba65b406dbaf1c812"
server_key_version: "encryptionkey@0"
I20260925 02:43:43.215406 6447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:43.232298 6447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:43.237636 6447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:43.237936 6447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:43.260461 6447 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:46463
I20260925 02:43:43.260488 6525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:46463 every 8 connection(s)
I20260925 02:43:43.261965 6447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:43:43.265838 6526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:43.266553 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 6447
I20260925 02:43:43.266775 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:43:43.267599 5346 external_mini_cluster.cc:1428] Setting key 4ddf22dfb51877c9c2eb58c508e9c15f
I20260925 02:43:43.280536 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68: Bootstrap starting.
Sep 25 02:43:43 dist-test-slave-n563 krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:43.284240 6526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:43.285497 6526 log.cc:824] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:43.288552 6526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68: No bootstrap required, opened a new log
I20260925 02:43:43.291363 6529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:43.269748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.5.56.190:46463)
0925 02:43:43.270684 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:43.270699 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:43.270745 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:43:43.272471 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:43.272495 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:43.272916 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:43.273261 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:43.275081 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:43.276365 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:43.278184 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:43.278923 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:43.279062 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:43.283771 (+ 4709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:43.283779 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:43.283795 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:43.283837 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:43.287079 (+ 3242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:43.287857 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:43.287864 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:43.287870 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:43.288045 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:43.288435 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:43.288440 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:43.288443 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:43.289027 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:43.289372 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:43.289803 (+ 431us) server_negotiation.cc:299] Negotiation successful
0925 02:43:43.290252 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":725,"thread_start_us":449,"threads_started":1}
I20260925 02:43:43.297181 6526 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ba619501044e739ae9cbc5f46a6c68" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46463 } }
I20260925 02:43:43.297628 6526 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:43.297744 6526 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30ba619501044e739ae9cbc5f46a6c68, State: Initialized, Role: FOLLOWER
I20260925 02:43:43.298333 6526 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ba619501044e739ae9cbc5f46a6c68" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46463 } }
I20260925 02:43:43.298655 6526 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:43:43.298784 6526 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:43:43.298956 6526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:43.301178 6526 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ba619501044e739ae9cbc5f46a6c68" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46463 } }
I20260925 02:43:43.301720 6526 leader_election.cc:302] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30ba619501044e739ae9cbc5f46a6c68; no voters:
I20260925 02:43:43.302649 6526 leader_election.cc:288] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:43:43.302793 6531 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:43.303915 6531 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [term 1 LEADER]: Becoming Leader. State: Replica: 30ba619501044e739ae9cbc5f46a6c68, State: Running, Role: LEADER
I20260925 02:43:43.304438 6531 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ba619501044e739ae9cbc5f46a6c68" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46463 } }
I20260925 02:43:43.305037 6526 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:43:43.307679 6532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30ba619501044e739ae9cbc5f46a6c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ba619501044e739ae9cbc5f46a6c68" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46463 } } }
I20260925 02:43:43.307654 6533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30ba619501044e739ae9cbc5f46a6c68. Latest consensus state: current_term: 1 leader_uuid: "30ba619501044e739ae9cbc5f46a6c68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30ba619501044e739ae9cbc5f46a6c68" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46463 } } }
I20260925 02:43:43.308136 6532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [sys.catalog]: This master's current role is: LEADER
I20260925 02:43:43.308135 6533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68 [sys.catalog]: This master's current role is: LEADER
I20260925 02:43:43.309717 6538 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:43:43.316226 6538 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:43:43.327962 6538 catalog_manager.cc:1347] Generated new cluster ID: dfece0d2728a478f8dcc427656871d9f
I20260925 02:43:43.328507 6538 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:43:43.345860 6538 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:43:43.347643 6538 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:43:43.354183 6538 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68: Generated new TSK 0
I20260925 02:43:43.355041 6538 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:43:43.399982 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:46463
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:43.738260 6552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:43.738659 6552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:43.738775 6552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:43.746449 6552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:43.746595 6552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:43.746687 6552 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:43:43.746773 6552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:43:43.756924 6552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.56.190:46463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:43.758371 6552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:43.759837 6552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:43.773864 6557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:43.773808 6558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:43.773933 6560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:43.774822 6552 server_base.cc:1034] running on GCE node
I20260925 02:43:43.775508 6552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:43.776572 6552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:43.777801 6552 hybrid_clock.cc:648] HybridClock initialized: now 1790304223777710 us; error 67 us; skew 500 ppm
I20260925 02:43:43.778137 6552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:43 dist-test-slave-n563 krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:43.786195 6552 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:43:43.787950 6552 webserver.cc:460] Webserver started at http://127.5.56.129:44657/ using document root <none> and password file <none>
I20260925 02:43:43.788689 6552 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:43.788781 6552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:43.789058 6552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:43.792387 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "8851b3d1d3b241838d3157f97317db19"
format_stamp: "Formatted at 2026-09-25 02:43:43 on dist-test-slave-n563"
server_key: "8a22bb79e73272ee1f330cddad205673"
server_key_iv: "09bd56f252fa3c8706746ba38111444c"
server_key_version: "encryptionkey@0"
I20260925 02:43:43.793170 6552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "8851b3d1d3b241838d3157f97317db19"
format_stamp: "Formatted at 2026-09-25 02:43:43 on dist-test-slave-n563"
server_key: "8a22bb79e73272ee1f330cddad205673"
server_key_iv: "09bd56f252fa3c8706746ba38111444c"
server_key_version: "encryptionkey@0"
I20260925 02:43:43.798638 6552 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 02:43:43.802273 6567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:43.803741 6552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:43:43.803932 6552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "8851b3d1d3b241838d3157f97317db19"
format_stamp: "Formatted at 2026-09-25 02:43:43 on dist-test-slave-n563"
server_key: "8a22bb79e73272ee1f330cddad205673"
server_key_iv: "09bd56f252fa3c8706746ba38111444c"
server_key_version: "encryptionkey@0"
I20260925 02:43:43.804262 6552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:43.821062 6552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:43.826041 6552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:43.826355 6552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:43.829361 6552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:43.829478 6552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:43.829663 6552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:43.829757 6552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:43:43 dist-test-slave-n563 krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:43.855319 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:43.830017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:48451 (local address 127.5.56.190:46463)
0925 02:43:43.830718 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:43.830725 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:43.831862 (+ 1137us) server_negotiation.cc:407] Connection header received
0925 02:43:43.833933 (+ 2071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:43.833942 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:43.834035 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:43.834172 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:43.836685 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:43.837855 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:43.840648 (+ 2793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:43.841265 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:43.841434 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:43.847427 (+ 5993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:43.847437 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:43.847444 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:43.847495 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:43.850358 (+ 2863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:43.851573 (+ 1215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:43.851581 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:43.851584 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:43.851752 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:43.852291 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:43.852297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:43.852301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:43.852631 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:43.852808 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:43.854401 (+ 1593us) server_negotiation.cc:299] Negotiation successful
0925 02:43:43.854757 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":436,"threads_started":1}
I20260925 02:43:43.858951 6577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:43.830718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:46463 (local address 127.5.56.129:48451)
0925 02:43:43.831671 (+ 953us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:43.831719 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:43.833641 (+ 1922us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:43.834494 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:43.834508 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:43.835390 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:43.836412 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:43.836449 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:43.838088 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:43.838094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:43.840482 (+ 2388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:43.840495 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:43.841650 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:43.841665 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:43.842092 (+ 427us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:43.843212 (+ 1120us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:43.843251 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:43.847083 (+ 3832us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:43.850591 (+ 3508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:43.850600 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:43.850618 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:43.851369 (+ 751us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:43.851889 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:43.851894 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:43.851897 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:43.852155 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:43.852805 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:43.852816 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:43.853466 (+ 650us) client_negotiation.cc:769] Sending connection context
0925 02:43:43.856049 (+ 2583us) client_negotiation.cc:240] Negotiation successful
0925 02:43:43.856830 (+ 781us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":394,"threads_started":1}
I20260925 02:43:43.887910 6552 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:34185
I20260925 02:43:43.887991 6689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:34185 every 8 connection(s)
I20260925 02:43:43.889261 6652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:43.869013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:46463 (local address 127.5.56.129:53623)
0925 02:43:43.869819 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:43.869839 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:43.869999 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:43.870454 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:43.870462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:43.871319 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:43.871812 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:43.871831 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:43.873798 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:43.873807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:43.876181 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:43.876207 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:43.877786 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:43.877797 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:43.878002 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:43.878841 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:43.878867 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:43.880335 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:43.885508 (+ 5173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:43.885513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:43.885517 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:43.886204 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:43.887007 (+ 803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:43.887015 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:43.887021 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:43.887213 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:43.888059 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:43.888064 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:43.888290 (+ 226us) client_negotiation.cc:769] Sending connection context
0925 02:43:43.888449 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 02:43:43.888813 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":453,"threads_started":1}
I20260925 02:43:43.889267 6576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:43.869590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:53623 (local address 127.5.56.190:46463)
0925 02:43:43.869838 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:43.869844 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:43.869933 (+ 89us) server_negotiation.cc:407] Connection header received
0925 02:43:43.870144 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:43.870152 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:43.870253 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:43.870368 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:43.872039 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:43.873443 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:43.876391 (+ 2948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:43.877193 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:43.877349 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:43.880637 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:43.880645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:43.880652 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:43.880701 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:43.885323 (+ 4622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:43.886366 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:43.886375 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:43.886416 (+ 41us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:43.886636 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:43.887386 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:43.887391 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:43.887394 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:43.887833 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:43.887988 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:43.888631 (+ 643us) server_negotiation.cc:299] Negotiation successful
0925 02:43:43.888897 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:43:43.889820 6552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:43:43.890340 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 6552
I20260925 02:43:43.891516 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:43:43.892388 5346 external_mini_cluster.cc:1428] Setting key a0089153cd1858c4351926f7870a7c59
I20260925 02:43:43.892005 6690 heartbeater.cc:347] Connected to a master server at 127.5.56.190:46463
I20260925 02:43:43.893513 6690 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:43.894512 6690 heartbeater.cc:511] Master 127.5.56.190:46463 requested a full tablet report, sending...
I20260925 02:43:43.896695 6479 ts_manager.cc:194] Registered new tserver with Master: 8851b3d1d3b241838d3157f97317db19 (127.5.56.129:34185)
I20260925 02:43:43.898312 6479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:48451
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:43:43.944053 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:46463
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:44.294762 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:44.295238 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:44.295467 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:44.311969 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:44.312166 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:44.312355 6694 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:43:44.312529 6694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:43:44.334270 6694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.56.190:46463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:44.336547 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:44.338577 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:44.356498 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:44.357169 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:44.357360 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:44.358542 6694 server_base.cc:1034] running on GCE node
I20260925 02:43:44.359395 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:44.360590 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:44.361989 6694 hybrid_clock.cc:648] HybridClock initialized: now 1790304224361881 us; error 82 us; skew 500 ppm
I20260925 02:43:44.362339 6694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:44 dist-test-slave-n563 krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304224, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:44.366288 6694 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:43:44.368129 6694 webserver.cc:460] Webserver started at http://127.5.56.130:40127/ using document root <none> and password file <none>
I20260925 02:43:44.369055 6694 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:44.369189 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:44.369545 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:44.372660 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "13d6d2c53d6b46fbb76e948a855d61e8"
format_stamp: "Formatted at 2026-09-25 02:43:44 on dist-test-slave-n563"
server_key: "bca3abed5dd99140d39256ece0d7a1d9"
server_key_iv: "bcde8cd2511ec9773b9d8118909e2402"
server_key_version: "encryptionkey@0"
I20260925 02:43:44.373497 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "13d6d2c53d6b46fbb76e948a855d61e8"
format_stamp: "Formatted at 2026-09-25 02:43:44 on dist-test-slave-n563"
server_key: "bca3abed5dd99140d39256ece0d7a1d9"
server_key_iv: "bcde8cd2511ec9773b9d8118909e2402"
server_key_version: "encryptionkey@0"
I20260925 02:43:44.378870 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:43:44.382738 6710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:44.384377 6694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:43:44.384578 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "13d6d2c53d6b46fbb76e948a855d61e8"
format_stamp: "Formatted at 2026-09-25 02:43:44 on dist-test-slave-n563"
server_key: "bca3abed5dd99140d39256ece0d7a1d9"
server_key_iv: "bcde8cd2511ec9773b9d8118909e2402"
server_key_version: "encryptionkey@0"
I20260925 02:43:44.384814 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:44.397405 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:44.401970 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:44.402251 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:44.405357 6694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:44.405479 6694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:44.405647 6694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:44.405735 6694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:43:44 dist-test-slave-n563 krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304224, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:44.442341 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:44.408543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:51181 (local address 127.5.56.190:46463)
0925 02:43:44.409082 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:44.409089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:44.409464 (+ 375us) server_negotiation.cc:407] Connection header received
0925 02:43:44.413178 (+ 3714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:44.413184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:44.413293 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:44.413453 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:44.416451 (+ 2998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.417966 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.422028 (+ 4062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.423002 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.423144 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:44.430798 (+ 7654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:44.430811 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:44.430819 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:44.430879 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:44.435997 (+ 5118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:44.437581 (+ 1584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:44.437592 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:44.437599 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:44.437797 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:44.438807 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:44.438813 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:44.438817 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:44.439496 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:44.439743 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:44.441356 (+ 1613us) server_negotiation.cc:299] Negotiation successful
0925 02:43:44.441697 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":257,"threads_started":1}
I20260925 02:43:44.445102 6719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:44.407743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:46463 (local address 127.5.56.130:51181)
0925 02:43:44.409302 (+ 1559us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:44.409341 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:44.412616 (+ 3275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:44.413852 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:44.413868 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:44.414790 (+ 922us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:44.416090 (+ 1300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:44.416141 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.418224 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.418234 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:44.421630 (+ 3396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:44.421664 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.423637 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.423649 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:44.424250 (+ 601us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:44.425467 (+ 1217us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:44.425514 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:44.430363 (+ 4849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:44.436297 (+ 5934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:44.436316 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:44.436353 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:44.437370 (+ 1017us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:44.438037 (+ 667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:44.438050 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:44.438057 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:44.438537 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:44.439933 (+ 1396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:44.439943 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:44.440747 (+ 804us) client_negotiation.cc:769] Sending connection context
0925 02:43:44.442467 (+ 1720us) client_negotiation.cc:240] Negotiation successful
0925 02:43:44.443107 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1298,"thread_start_us":1054,"threads_started":1}
I20260925 02:43:44.460652 6694 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:35521
I20260925 02:43:44.460781 6831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:35521 every 8 connection(s)
I20260925 02:43:44.462508 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:43:44.463212 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 6694
I20260925 02:43:44.463486 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:43:44.464140 5346 external_mini_cluster.cc:1428] Setting key 968981c777f3bb6af9b87cc6cafd8bf3
I20260925 02:43:44.470942 6832 heartbeater.cc:347] Connected to a master server at 127.5.56.190:46463
I20260925 02:43:44.471210 6832 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:44.472146 6832 heartbeater.cc:511] Master 127.5.56.190:46463 requested a full tablet report, sending...
I20260925 02:43:44.473932 6479 ts_manager.cc:194] Registered new tserver with Master: 13d6d2c53d6b46fbb76e948a855d61e8 (127.5.56.130:35521)
I20260925 02:43:44.475595 6479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:51181
I20260925 02:43:44.476056 6819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:44.453630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:46463 (local address 127.5.56.130:44171)
0925 02:43:44.455823 (+ 2193us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:44.455855 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:44.456169 (+ 314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:44.456866 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:44.456874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:44.457740 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:44.458319 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:44.458339 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.460168 (+ 1829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.460173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:44.461850 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:44.461866 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.463585 (+ 1719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.463596 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:44.463874 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:44.464752 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:44.464787 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:44.466301 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:44.471741 (+ 5440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:44.471750 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:44.471758 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:44.472513 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:44.473085 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:44.473093 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:44.473100 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:44.473291 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:44.474348 (+ 1057us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:44.474357 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:44.474665 (+ 308us) client_negotiation.cc:769] Sending connection context
0925 02:43:44.474876 (+ 211us) client_negotiation.cc:240] Negotiation successful
0925 02:43:44.475356 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1906,"thread_start_us":1692,"threads_started":1}
I20260925 02:43:44.477396 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:44.453814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:44171 (local address 127.5.56.190:46463)
0925 02:43:44.454096 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:44.454106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:44.455996 (+ 1890us) server_negotiation.cc:407] Connection header received
0925 02:43:44.456365 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:44.456374 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:44.456482 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:44.456569 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:44.458610 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.459961 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.462043 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.463041 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.463209 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:44.466646 (+ 3437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:44.466657 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:44.466665 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:44.466718 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:44.471530 (+ 4812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:44.472696 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:44.472706 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:44.472713 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:44.472908 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:44.473445 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:44.473454 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:44.473460 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:44.474034 (+ 574us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:44.474260 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:44.476578 (+ 2318us) server_negotiation.cc:299] Negotiation successful
0925 02:43:44.476910 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:43:44.522703 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:46463
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:44.864477 6836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:44.864843 6836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:44.865027 6836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:44.873392 6836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:44.873586 6836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:44.873766 6836 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:43:44.873920 6836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:43:44.884531 6836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.56.190:46463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:44.885962 6836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:44.887657 6836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:44.902014 6842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:44.902047 6844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:44.901979 6841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:44.902568 6690 heartbeater.cc:503] Master 127.5.56.190:46463 was elected leader, sending a full tablet report...
I20260925 02:43:44.902652 6836 server_base.cc:1034] running on GCE node
I20260925 02:43:44.903668 6836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:44.904826 6836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:44.906086 6836 hybrid_clock.cc:648] HybridClock initialized: now 1790304224906010 us; error 51 us; skew 500 ppm
I20260925 02:43:44.906490 6836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:44 dist-test-slave-n563 krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304224, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:44.910164 6836 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:43:44.911779 6836 webserver.cc:460] Webserver started at http://127.5.56.131:42623/ using document root <none> and password file <none>
I20260925 02:43:44.912552 6836 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:44.912645 6836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:44.912925 6836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:44.915989 6836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "8a4290762e134c4da9871393d120cb6e"
format_stamp: "Formatted at 2026-09-25 02:43:44 on dist-test-slave-n563"
server_key: "feb7a4e7251522caec920dd00c8ed439"
server_key_iv: "a55d5322678b5071d9c1efad01242543"
server_key_version: "encryptionkey@0"
I20260925 02:43:44.916754 6836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "8a4290762e134c4da9871393d120cb6e"
format_stamp: "Formatted at 2026-09-25 02:43:44 on dist-test-slave-n563"
server_key: "feb7a4e7251522caec920dd00c8ed439"
server_key_iv: "a55d5322678b5071d9c1efad01242543"
server_key_version: "encryptionkey@0"
I20260925 02:43:44.922278 6836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:43:44.925974 6851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:44.927495 6836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 02:43:44.927716 6836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "8a4290762e134c4da9871393d120cb6e"
format_stamp: "Formatted at 2026-09-25 02:43:44 on dist-test-slave-n563"
server_key: "feb7a4e7251522caec920dd00c8ed439"
server_key_iv: "a55d5322678b5071d9c1efad01242543"
server_key_version: "encryptionkey@0"
I20260925 02:43:44.927954 6836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:44.951166 6836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:44.955633 6836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:44.955940 6836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:44.958731 6836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:44.958854 6836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:44.959034 6836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:44.959131 6836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:43:44 dist-test-slave-n563 krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304224, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:44.985337 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:44.959662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:37533 (local address 127.5.56.190:46463)
0925 02:43:44.959877 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:44.959883 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:44.961399 (+ 1516us) server_negotiation.cc:407] Connection header received
0925 02:43:44.964061 (+ 2662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:44.964068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:44.964180 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:44.964309 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:44.966883 (+ 2574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.967997 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.970869 (+ 2872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.971501 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.971676 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:44.976646 (+ 4970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:44.976656 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:44.976664 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:44.976723 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:44.981350 (+ 4627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:44.982560 (+ 1210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:44.982573 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:44.982576 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:44.982768 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:44.983337 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:44.983343 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:44.983347 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:44.983682 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:44.983849 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:44.984707 (+ 858us) server_negotiation.cc:299] Negotiation successful
0925 02:43:44.984947 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 02:43:44.986778 6860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:44.960228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:46463 (local address 127.5.56.131:37533)
0925 02:43:44.961185 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:44.961246 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:44.963539 (+ 2293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:44.964626 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:44.964642 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:44.965586 (+ 944us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:44.966646 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:44.966688 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.968263 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.968270 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:44.970702 (+ 2432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:44.970720 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:44.971789 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:44.971796 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:44.972047 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:44.972849 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:44.972873 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:44.976256 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:44.981567 (+ 5311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:44.981583 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:44.981627 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:44.982381 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:44.982937 (+ 556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:44.982944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:44.982953 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:44.983199 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:44.983821 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:44.983830 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:44.984293 (+ 463us) client_negotiation.cc:769] Sending connection context
0925 02:43:44.984599 (+ 306us) client_negotiation.cc:240] Negotiation successful
0925 02:43:44.985193 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":415,"threads_started":1}
I20260925 02:43:45.011927 6836 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:43561
I20260925 02:43:45.012033 6971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:43561 every 8 connection(s)
I20260925 02:43:45.013487 6836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:43:45.015144 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 6836
I20260925 02:43:45.015359 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:43:45.015738 6972 heartbeater.cc:347] Connected to a master server at 127.5.56.190:46463
I20260925 02:43:45.015969 6972 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:45.015978 5346 external_mini_cluster.cc:1428] Setting key d49d8ecd0f3f08e0c6b827fa26a4fe13
I20260925 02:43:45.016702 6972 heartbeater.cc:511] Master 127.5.56.190:46463 requested a full tablet report, sending...
I20260925 02:43:45.018309 6479 ts_manager.cc:194] Registered new tserver with Master: 8a4290762e134c4da9871393d120cb6e (127.5.56.131:43561)
I20260925 02:43:45.019678 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:43:45.019639 6479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:37533
I20260925 02:43:45.023958 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.005369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:39003 (local address 127.5.56.190:46463)
0925 02:43:45.005605 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:45.005613 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:45.006687 (+ 1074us) server_negotiation.cc:407] Connection header received
0925 02:43:45.006912 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:45.006920 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:45.007023 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:45.007131 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:45.008920 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.009905 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.011835 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.012474 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.012621 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.016888 (+ 4267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:45.016894 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:45.016899 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:45.016960 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:45.021054 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:45.021961 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:45.021967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:45.021970 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:45.022146 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:45.022650 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:45.022655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:45.022659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:45.022929 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:45.023041 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:45.023544 (+ 503us) server_negotiation.cc:299] Negotiation successful
0925 02:43:45.023724 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:43:45.024408 6958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.005662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:46463 (local address 127.5.56.131:39003)
0925 02:43:45.006560 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:45.006586 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:45.006764 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:45.007320 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:45.007323 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:45.008121 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:45.008689 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.008707 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.010049 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.010057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.011579 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.011618 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.012683 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.012688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.012817 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.013589 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:45.013606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:45.016681 (+ 3075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:45.021207 (+ 4526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:45.021212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:45.021217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:45.021798 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:45.022282 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:45.022290 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:45.022297 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:45.022509 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:45.023064 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:45.023071 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:45.023287 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 02:43:45.023450 (+ 163us) client_negotiation.cc:240] Negotiation successful
0925 02:43:45.023832 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":516,"threads_started":1}
Sep 25 02:43:45 dist-test-slave-n563 krb5kdc[6434](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:43:45 dist-test-slave-n563 krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:45.070609 6720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.055181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.5.56.190:46463)
0925 02:43:45.055374 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:45.055378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:45.055757 (+ 379us) server_negotiation.cc:407] Connection header received
0925 02:43:45.055914 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:45.055918 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:45.055984 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:45.056092 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:45.057434 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.058181 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.059820 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.060236 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.060333 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.065154 (+ 4821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:45.065161 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:45.065164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:45.065202 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:45.067941 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:45.068827 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:45.068832 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:45.068834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:45.068962 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:45.069359 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:45.069363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:45.069365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:45.069650 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:45.069855 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:45.070105 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 02:43:45.070269 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:43:45.074769 6479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:43:45.081743 6479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:43:45.107524 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.095440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34185 (local address 127.0.0.1:52014)
0925 02:43:45.097727 (+ 2287us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:45.097744 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:45.097861 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:45.098894 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:45.098901 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:45.098931 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:45.099402 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.099419 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.101601 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.101608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.104332 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.104348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.106336 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.106342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.106608 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.106713 (+ 105us) client_negotiation.cc:769] Sending connection context
0925 02:43:45.106842 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 02:43:45.107050 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2157,"mutex_wait_us":24,"thread_start_us":1106,"threads_started":1}
I20260925 02:43:45.108764 6986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.095393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:43561 (local address 127.0.0.1:60290)
0925 02:43:45.096959 (+ 1566us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:45.096996 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:45.097166 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:45.101831 (+ 4665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:45.101838 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:45.101864 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:45.102491 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.102502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.103974 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.103979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.105881 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.105893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.107887 (+ 1994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.107893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.108064 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.108102 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:43:45.108165 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 02:43:45.108291 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1330,"mutex_wait_us":27,"thread_start_us":2066,"threads_started":1}
I20260925 02:43:45.109161 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.095891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52014 (local address 127.5.56.129:34185)
0925 02:43:45.096558 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:45.096570 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:45.098006 (+ 1436us) server_negotiation.cc:407] Connection header received
0925 02:43:45.098124 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:45.098132 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:45.098424 (+ 292us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:45.098624 (+ 200us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:45.099622 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.101453 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.104526 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.106066 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.106218 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.106467 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:45.108084 (+ 1617us) server_negotiation.cc:299] Negotiation successful
0925 02:43:45.108293 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":272,"threads_started":1}
I20260925 02:43:45.109426 6989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.098091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:35521 (local address 127.0.0.1:36926)
0925 02:43:45.098750 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:45.098773 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:45.098921 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:45.101659 (+ 2738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:45.101668 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:45.101696 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:45.102483 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.102504 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.104395 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.104403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.106688 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.106700 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.108512 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.108520 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.108715 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.108757 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:43:45.108830 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:43:45.108978 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":80,"spinlock_wait_cycles":1920,"thread_start_us":701,"threads_started":1}
I20260925 02:43:45.110153 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.098606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36926 (local address 127.5.56.130:35521)
0925 02:43:45.100097 (+ 1491us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:45.100104 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:45.100157 (+ 53us) server_negotiation.cc:407] Connection header received
0925 02:43:45.100250 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:45.100259 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:45.100525 (+ 266us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:45.100715 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:45.102748 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.104231 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.106847 (+ 2616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.107952 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.108093 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.108357 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:45.109728 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0925 02:43:45.109924 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1324,"thread_start_us":1204,"threads_started":1}
I20260925 02:43:45.111078 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.096070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60290 (local address 127.5.56.131:43561)
0925 02:43:45.101371 (+ 5301us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:45.101387 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:45.101412 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:43:45.101494 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:45.101499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:45.101650 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:45.101844 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:45.102707 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.103771 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.106105 (+ 2334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.107653 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.109210 (+ 1557us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.110501 (+ 1291us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:45.110580 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 02:43:45.110740 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5136,"thread_start_us":4935,"threads_started":1}
I20260925 02:43:45.113332 6761 tablet_service.cc:1467] Processing CreateTablet for tablet acc99cf4f4114678b4f3f9c9f8485624 (DEFAULT_TABLE table=test-table [id=bb2da58080274feb8a00b1106331c2a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:45.113379 6619 tablet_service.cc:1467] Processing CreateTablet for tablet acc99cf4f4114678b4f3f9c9f8485624 (DEFAULT_TABLE table=test-table [id=bb2da58080274feb8a00b1106331c2a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:45.114776 6761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc99cf4f4114678b4f3f9c9f8485624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:45.114789 6619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc99cf4f4114678b4f3f9c9f8485624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:45.114495 6901 tablet_service.cc:1467] Processing CreateTablet for tablet acc99cf4f4114678b4f3f9c9f8485624 (DEFAULT_TABLE table=test-table [id=bb2da58080274feb8a00b1106331c2a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:45.115830 6901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet acc99cf4f4114678b4f3f9c9f8485624. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:45.127019 6991 tablet_bootstrap.cc:492] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: Bootstrap starting.
I20260925 02:43:45.129387 6992 tablet_bootstrap.cc:492] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19: Bootstrap starting.
I20260925 02:43:45.130821 6991 tablet_bootstrap.cc:654] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:45.130980 6993 tablet_bootstrap.cc:492] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: Bootstrap starting.
I20260925 02:43:45.132228 6991 log.cc:824] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:45.133725 6992 tablet_bootstrap.cc:654] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:45.134641 6993 tablet_bootstrap.cc:654] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:45.135082 6992 log.cc:824] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:45.135702 6993 log.cc:824] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:45.136179 6991 tablet_bootstrap.cc:492] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: No bootstrap required, opened a new log
I20260925 02:43:45.136511 6991 ts_tablet_manager.cc:1397] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.007s
I20260925 02:43:45.138227 6992 tablet_bootstrap.cc:492] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19: No bootstrap required, opened a new log
I20260925 02:43:45.138543 6992 ts_tablet_manager.cc:1397] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 02:43:45.140241 6993 tablet_bootstrap.cc:492] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: No bootstrap required, opened a new log
I20260925 02:43:45.140513 6993 ts_tablet_manager.cc:1397] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260925 02:43:45.147687 6991 raft_consensus.cc:348] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } }
I20260925 02:43:45.148195 6991 raft_consensus.cc:374] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:45.147969 6993 raft_consensus.cc:348] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } }
I20260925 02:43:45.148298 6991 raft_consensus.cc:729] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d6d2c53d6b46fbb76e948a855d61e8, State: Initialized, Role: FOLLOWER
I20260925 02:43:45.148341 6993 raft_consensus.cc:374] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:45.148428 6993 raft_consensus.cc:729] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4290762e134c4da9871393d120cb6e, State: Initialized, Role: FOLLOWER
I20260925 02:43:45.149025 6993 consensus_queue.cc:260] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } }
I20260925 02:43:45.149044 6991 consensus_queue.cc:260] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } }
I20260925 02:43:45.150265 6972 heartbeater.cc:503] Master 127.5.56.190:46463 was elected leader, sending a full tablet report...
I20260925 02:43:45.150749 6832 heartbeater.cc:503] Master 127.5.56.190:46463 was elected leader, sending a full tablet report...
I20260925 02:43:45.151165 6992 raft_consensus.cc:348] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } }
I20260925 02:43:45.151698 6992 raft_consensus.cc:374] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:45.152169 6992 raft_consensus.cc:729] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8851b3d1d3b241838d3157f97317db19, State: Initialized, Role: FOLLOWER
I20260925 02:43:45.152683 6991 ts_tablet_manager.cc:1428] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: Time spent starting tablet: real 0.016s user 0.016s sys 0.001s
I20260925 02:43:45.153013 6992 consensus_queue.cc:260] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } }
I20260925 02:43:45.153365 6993 ts_tablet_manager.cc:1428] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260925 02:43:45.155346 6992 ts_tablet_manager.cc:1428] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20260925 02:43:45.193450 6998 raft_consensus.cc:482] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:43:45.193821 6998 raft_consensus.cc:504] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } }
I20260925 02:43:45.195684 6998 leader_election.cc:288] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a4290762e134c4da9871393d120cb6e (127.5.56.131:43561), 8851b3d1d3b241838d3157f97317db19 (127.5.56.129:34185)
I20260925 02:43:45.203306 6987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.196224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:59003 (local address 127.5.56.129:34185)
0925 02:43:45.196430 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:45.196436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:45.197382 (+ 946us) server_negotiation.cc:407] Connection header received
0925 02:43:45.197594 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:45.197598 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:45.197670 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:45.197772 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:45.198444 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.199155 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.201190 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.202268 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.202468 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.202556 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:45.202884 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 02:43:45.203018 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:43:45.203315 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.196052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:35801 (local address 127.5.56.131:43561)
0925 02:43:45.196279 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:45.196288 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:45.197230 (+ 942us) server_negotiation.cc:407] Connection header received
0925 02:43:45.197465 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:45.197470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:45.197538 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:45.197637 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:45.198232 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.199042 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.201055 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.202110 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.202284 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.202381 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:45.202905 (+ 524us) server_negotiation.cc:299] Negotiation successful
0925 02:43:45.203044 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:43:45.203471 7001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.196275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34185 (local address 127.5.56.130:59003)
0925 02:43:45.197266 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:45.197280 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:45.197445 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:45.197794 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:45.197798 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:45.197837 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:45.198267 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.198275 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.199298 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.199310 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.200923 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.200937 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.202517 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.202523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.202698 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.202743 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:43:45.202855 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:43:45.203010 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":900,"mutex_wait_us":29,"thread_start_us":628,"threads_started":1}
I20260925 02:43:45.203950 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.196105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:43561 (local address 127.5.56.130:35801)
0925 02:43:45.197112 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:45.197129 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:45.197314 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:45.197653 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:45.197657 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:45.197691 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:45.198084 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.198093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.199173 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.199181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.200880 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.200900 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.202572 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.202576 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.202701 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.202772 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 02:43:45.202879 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:43:45.203014 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"mutex_wait_us":83,"thread_start_us":565,"threads_started":1}
I20260925 02:43:45.204577 6639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc99cf4f4114678b4f3f9c9f8485624" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8851b3d1d3b241838d3157f97317db19" is_pre_election: true
I20260925 02:43:45.204928 6923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc99cf4f4114678b4f3f9c9f8485624" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4290762e134c4da9871393d120cb6e" is_pre_election: true
I20260925 02:43:45.205212 6639 raft_consensus.cc:2454] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 0.
I20260925 02:43:45.205477 6923 raft_consensus.cc:2454] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 0.
I20260925 02:43:45.205993 6714 leader_election.cc:302] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d6d2c53d6b46fbb76e948a855d61e8, 8851b3d1d3b241838d3157f97317db19; no voters:
I20260925 02:43:45.206490 6998 raft_consensus.cc:2781] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:43:45.206664 6998 raft_consensus.cc:482] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:43:45.206787 6998 raft_consensus.cc:3037] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:45.209282 6998 raft_consensus.cc:504] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } }
I20260925 02:43:45.209931 6998 leader_election.cc:288] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 election: Requested vote from peers 8a4290762e134c4da9871393d120cb6e (127.5.56.131:43561), 8851b3d1d3b241838d3157f97317db19 (127.5.56.129:34185)
I20260925 02:43:45.210569 6639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc99cf4f4114678b4f3f9c9f8485624" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8851b3d1d3b241838d3157f97317db19"
I20260925 02:43:45.210567 6923 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "acc99cf4f4114678b4f3f9c9f8485624" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4290762e134c4da9871393d120cb6e"
I20260925 02:43:45.210807 6639 raft_consensus.cc:3037] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:45.210812 6923 raft_consensus.cc:3037] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:45.214154 6923 raft_consensus.cc:2454] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 1.
I20260925 02:43:45.214157 6639 raft_consensus.cc:2454] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 1.
I20260925 02:43:45.214784 6712 leader_election.cc:302] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d6d2c53d6b46fbb76e948a855d61e8, 8a4290762e134c4da9871393d120cb6e; no voters:
I20260925 02:43:45.215858 6998 raft_consensus.cc:2781] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:45.216692 6998 raft_consensus.cc:686] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 LEADER]: Becoming Leader. State: Replica: 13d6d2c53d6b46fbb76e948a855d61e8, State: Running, Role: LEADER
I20260925 02:43:45.217264 6998 consensus_queue.cc:237] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } }
I20260925 02:43:45.222846 6477 catalog_manager.cc:5495] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d6d2c53d6b46fbb76e948a855d61e8 (127.5.56.130). New cstate: current_term: 1 leader_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } health_report { overall_health: HEALTHY } } }
W20260925 02:43:45.229739 6833 tablet.cc:2369] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:43:45.254750 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.248929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36934 (local address 127.5.56.130:35521)
0925 02:43:45.249118 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:45.249144 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:45.249298 (+ 154us) server_negotiation.cc:407] Connection header received
0925 02:43:45.249637 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:45.249641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:45.249726 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:45.249853 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:43:45.250285 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.251183 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.252609 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.253075 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.253173 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.253627 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:43:45.254182 (+ 555us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:43:45.254296 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:45.254453 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 02:43:45.254526 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:43:45.263633 6639 raft_consensus.cc:1261] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19 [term 1 FOLLOWER]: Refusing update from remote peer 13d6d2c53d6b46fbb76e948a855d61e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:45.264927 6998 consensus_queue.cc:1035] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:43:45.265239 6973 tablet.cc:2369] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:43:45.266112 6923 raft_consensus.cc:1261] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e [term 1 FOLLOWER]: Refusing update from remote peer 13d6d2c53d6b46fbb76e948a855d61e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:45.266992 6998 consensus_queue.cc:1035] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:45.279780 7005 mvcc.cc:205] Tried to move back new op lower bound from 7333086106667933696 to 7333086106494193664. Current Snapshot: MvccSnapshot[applied={T|T < 7333086106667933696}]
I20260925 02:43:45.280349 7006 mvcc.cc:205] Tried to move back new op lower bound from 7333086106667933696 to 7333086106494193664. Current Snapshot: MvccSnapshot[applied={T|T < 7333086106667933696}]
I20260925 02:43:45.306052 6477 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:45.306424 6477 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:45.310434 6477 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68: Sending DeleteTablet for 3 replicas of tablet acc99cf4f4114678b4f3f9c9f8485624
I20260925 02:43:45.311900 6761 tablet_service.cc:1514] Processing DeleteTablet for tablet acc99cf4f4114678b4f3f9c9f8485624 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:45 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:36926
I20260925 02:43:45.312849 7015 tablet_replica.cc:330] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: stopping tablet replica
I20260925 02:43:45.313498 7015 raft_consensus.cc:2229] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:43:45.314069 6477 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 02:43:45.314267 7015 raft_consensus.cc:2258] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 02:43:45.314664 6477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:43:45.314754 6901 tablet_service.cc:1514] Processing DeleteTablet for tablet acc99cf4f4114678b4f3f9c9f8485624 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:45 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:60290
I20260925 02:43:45.315553 6619 tablet_service.cc:1514] Processing DeleteTablet for tablet acc99cf4f4114678b4f3f9c9f8485624 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:45 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:52014
I20260925 02:43:45.316952 7015 ts_tablet_manager.cc:1905] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:45.317669 7017 tablet_replica.cc:330] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19: stopping tablet replica
I20260925 02:43:45.318131 7017 raft_consensus.cc:2229] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:45.318873 7016 tablet_replica.cc:330] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: stopping tablet replica
I20260925 02:43:45.319330 7016 raft_consensus.cc:2229] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:45.319828 7017 raft_consensus.cc:2258] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:45.320534 7016 raft_consensus.cc:2258] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:45.323570 7017 ts_tablet_manager.cc:1905] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:45.324296 6761 tablet_service.cc:1467] Processing CreateTablet for tablet d0b0db2750a54370a1513470119121b8 (DEFAULT_TABLE table=test-table [id=dd5b46db5d1649cfb479390341ec838a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:45.324803 6761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b0db2750a54370a1513470119121b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:45.327044 6900 tablet_service.cc:1467] Processing CreateTablet for tablet d0b0db2750a54370a1513470119121b8 (DEFAULT_TABLE table=test-table [id=dd5b46db5d1649cfb479390341ec838a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:45.327627 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b0db2750a54370a1513470119121b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:45.330482 7017 ts_tablet_manager.cc:1918] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:43:45.330646 7017 log.cc:1196] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/acc99cf4f4114678b4f3f9c9f8485624
I20260925 02:43:45.331251 7017 ts_tablet_manager.cc:1939] T acc99cf4f4114678b4f3f9c9f8485624 P 8851b3d1d3b241838d3157f97317db19: Deleting consensus metadata
I20260925 02:43:45.331157 6619 tablet_service.cc:1467] Processing CreateTablet for tablet d0b0db2750a54370a1513470119121b8 (DEFAULT_TABLE table=test-table [id=dd5b46db5d1649cfb479390341ec838a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:45.331635 6619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0b0db2750a54370a1513470119121b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:45.333886 6991 tablet_bootstrap.cc:492] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8: Bootstrap starting.
I20260925 02:43:45.333339 6466 catalog_manager.cc:4841] TS 8851b3d1d3b241838d3157f97317db19 (127.5.56.129:34185): tablet acc99cf4f4114678b4f3f9c9f8485624 (table test-table [id=bb2da58080274feb8a00b1106331c2a4]) successfully deleted
I20260925 02:43:45.335240 7016 ts_tablet_manager.cc:1905] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:45.335659 6991 tablet_bootstrap.cc:654] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:45.338557 6993 tablet_bootstrap.cc:492] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e: Bootstrap starting.
I20260925 02:43:45.340301 6993 tablet_bootstrap.cc:654] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:45.341599 6992 tablet_bootstrap.cc:492] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19: Bootstrap starting.
I20260925 02:43:45.343245 6992 tablet_bootstrap.cc:654] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:45.343333 7016 ts_tablet_manager.cc:1918] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:43:45.343499 7016 log.cc:1196] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/acc99cf4f4114678b4f3f9c9f8485624
I20260925 02:43:45.344101 7016 ts_tablet_manager.cc:1939] T acc99cf4f4114678b4f3f9c9f8485624 P 8a4290762e134c4da9871393d120cb6e: Deleting consensus metadata
I20260925 02:43:45.346359 6464 catalog_manager.cc:4841] TS 8a4290762e134c4da9871393d120cb6e (127.5.56.131:43561): tablet acc99cf4f4114678b4f3f9c9f8485624 (table test-table [id=bb2da58080274feb8a00b1106331c2a4]) successfully deleted
I20260925 02:43:45.351773 7015 ts_tablet_manager.cc:1918] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:43:45.352033 7015 log.cc:1196] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/acc99cf4f4114678b4f3f9c9f8485624
I20260925 02:43:45.352375 6993 tablet_bootstrap.cc:492] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e: No bootstrap required, opened a new log
I20260925 02:43:45.352571 6993 ts_tablet_manager.cc:1397] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20260925 02:43:45.353056 6992 tablet_bootstrap.cc:492] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19: No bootstrap required, opened a new log
I20260925 02:43:45.353247 6992 ts_tablet_manager.cc:1397] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20260925 02:43:45.353734 6993 raft_consensus.cc:348] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.354051 6993 raft_consensus.cc:374] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:45.353958 6992 raft_consensus.cc:348] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.354161 6993 raft_consensus.cc:729] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4290762e134c4da9871393d120cb6e, State: Initialized, Role: FOLLOWER
I20260925 02:43:45.354235 6992 raft_consensus.cc:374] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:45.354351 6992 raft_consensus.cc:729] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8851b3d1d3b241838d3157f97317db19, State: Initialized, Role: FOLLOWER
I20260925 02:43:45.354689 6992 consensus_queue.cc:260] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.354604 6993 consensus_queue.cc:260] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.355268 6992 ts_tablet_manager.cc:1428] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:43:45.355876 7015 ts_tablet_manager.cc:1939] T acc99cf4f4114678b4f3f9c9f8485624 P 13d6d2c53d6b46fbb76e948a855d61e8: Deleting consensus metadata
I20260925 02:43:45.355991 6993 ts_tablet_manager.cc:1428] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 02:43:45.358556 6465 catalog_manager.cc:4841] TS 13d6d2c53d6b46fbb76e948a855d61e8 (127.5.56.130:35521): tablet acc99cf4f4114678b4f3f9c9f8485624 (table test-table [id=bb2da58080274feb8a00b1106331c2a4]) successfully deleted
I20260925 02:43:45.359973 6991 tablet_bootstrap.cc:492] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8: No bootstrap required, opened a new log
I20260925 02:43:45.360162 6991 ts_tablet_manager.cc:1397] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.004s
I20260925 02:43:45.361157 6991 raft_consensus.cc:348] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.361428 6991 raft_consensus.cc:374] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:45.361541 6991 raft_consensus.cc:729] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d6d2c53d6b46fbb76e948a855d61e8, State: Initialized, Role: FOLLOWER
I20260925 02:43:45.361831 6991 consensus_queue.cc:260] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.362560 6991 ts_tablet_manager.cc:1428] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
W20260925 02:43:45.392880 6691 tablet.cc:2369] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:43:45.605775 6998 raft_consensus.cc:482] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:43:45.606029 6998 raft_consensus.cc:504] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.606853 6998 leader_election.cc:288] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8851b3d1d3b241838d3157f97317db19 (127.5.56.129:34185), 8a4290762e134c4da9871393d120cb6e (127.5.56.131:43561)
I20260925 02:43:45.607471 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b0db2750a54370a1513470119121b8" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4290762e134c4da9871393d120cb6e" is_pre_election: true
I20260925 02:43:45.607569 6639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b0db2750a54370a1513470119121b8" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8851b3d1d3b241838d3157f97317db19" is_pre_election: true
I20260925 02:43:45.607723 6920 raft_consensus.cc:2454] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 0.
I20260925 02:43:45.607934 6639 raft_consensus.cc:2454] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 0.
I20260925 02:43:45.608156 6712 leader_election.cc:302] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d6d2c53d6b46fbb76e948a855d61e8, 8a4290762e134c4da9871393d120cb6e; no voters:
I20260925 02:43:45.608484 6998 raft_consensus.cc:2781] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:43:45.608609 6998 raft_consensus.cc:482] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:43:45.608738 6998 raft_consensus.cc:3037] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:45.611093 6998 raft_consensus.cc:504] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.611829 6998 leader_election.cc:288] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 election: Requested vote from peers 8851b3d1d3b241838d3157f97317db19 (127.5.56.129:34185), 8a4290762e134c4da9871393d120cb6e (127.5.56.131:43561)
I20260925 02:43:45.612332 6639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b0db2750a54370a1513470119121b8" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8851b3d1d3b241838d3157f97317db19"
I20260925 02:43:45.612409 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0b0db2750a54370a1513470119121b8" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4290762e134c4da9871393d120cb6e"
I20260925 02:43:45.612571 6639 raft_consensus.cc:3037] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:45.612620 6920 raft_consensus.cc:3037] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:45.615620 6920 raft_consensus.cc:2454] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 1.
I20260925 02:43:45.615629 6639 raft_consensus.cc:2454] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 1.
I20260925 02:43:45.616081 6714 leader_election.cc:302] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d6d2c53d6b46fbb76e948a855d61e8, 8851b3d1d3b241838d3157f97317db19; no voters:
I20260925 02:43:45.616394 6998 raft_consensus.cc:2781] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:45.616621 6998 raft_consensus.cc:686] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 LEADER]: Becoming Leader. State: Replica: 13d6d2c53d6b46fbb76e948a855d61e8, State: Running, Role: LEADER
I20260925 02:43:45.616890 6998 consensus_queue.cc:237] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.619755 6477 catalog_manager.cc:5495] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d6d2c53d6b46fbb76e948a855d61e8 (127.5.56.130). New cstate: current_term: 1 leader_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } health_report { overall_health: UNKNOWN } } }
Sep 25 02:43:45 dist-test-slave-n563 krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304223, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:45.693192 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.676831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:46463 (local address 127.0.0.1:59516)
0925 02:43:45.677818 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:45.677835 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:45.677973 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:45.678336 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:45.678339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:45.679000 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:45.679211 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.679220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.680862 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.680866 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.681789 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:45.681800 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.682892 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.682896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:45.683051 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.683789 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:45.683806 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:45.686025 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:45.690617 (+ 4592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:45.690629 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:45.690635 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:45.691052 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:45.691525 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:45.691528 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:45.691531 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:45.691669 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:45.692363 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:45.692369 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:45.692514 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 02:43:45.692663 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 02:43:45.692865 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":825,"mutex_wait_us":1,"thread_start_us":690,"threads_started":1}
I20260925 02:43:45.693646 7028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:45.676961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59516 (local address 127.5.56.190:46463)
0925 02:43:45.677653 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:45.677663 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:45.677963 (+ 300us) server_negotiation.cc:407] Connection header received
0925 02:43:45.678079 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:45.678086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:45.678176 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:45.678318 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:45.679376 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.680725 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.681951 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:45.682726 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:45.682880 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:45.686262 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:45.686274 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:45.686279 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:45.686321 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:45.690471 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:45.691208 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:45.691214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:45.691218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:45.691398 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:45.691788 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:45.691794 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:45.691798 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:45.692217 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:45.692395 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:45.692819 (+ 424us) server_negotiation.cc:299] Negotiation successful
0925 02:43:45.693098 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":32,"server-negotiator.queue_time_us":508,"thread_start_us":330,"threads_started":1}
I20260925 02:43:45.699196 6477 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:59516:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:43:45.700335 6477 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:43:45.708087 6619 tablet_service.cc:1467] Processing CreateTablet for tablet fff49ff87b10430b8d6f604a352fd605 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c2b9b29783c54f9d9880e76943f62f1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:43:45.708112 6761 tablet_service.cc:1467] Processing CreateTablet for tablet fff49ff87b10430b8d6f604a352fd605 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c2b9b29783c54f9d9880e76943f62f1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:43:45.708698 6619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff49ff87b10430b8d6f604a352fd605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:45.708555 6900 tablet_service.cc:1467] Processing CreateTablet for tablet fff49ff87b10430b8d6f604a352fd605 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c2b9b29783c54f9d9880e76943f62f1c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:43:45.709141 6900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff49ff87b10430b8d6f604a352fd605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:45.708693 6761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff49ff87b10430b8d6f604a352fd605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:45.716832 6992 tablet_bootstrap.cc:492] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19: Bootstrap starting.
I20260925 02:43:45.717381 6993 tablet_bootstrap.cc:492] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e: Bootstrap starting.
I20260925 02:43:45.718706 6992 tablet_bootstrap.cc:654] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:45.719246 6993 tablet_bootstrap.cc:654] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:45.720428 6991 tablet_bootstrap.cc:492] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: Bootstrap starting.
I20260925 02:43:45.722673 6991 tablet_bootstrap.cc:654] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:45.725737 6993 tablet_bootstrap.cc:492] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e: No bootstrap required, opened a new log
I20260925 02:43:45.725948 6993 ts_tablet_manager.cc:1397] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20260925 02:43:45.727245 6991 tablet_bootstrap.cc:492] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: No bootstrap required, opened a new log
I20260925 02:43:45.727403 6992 tablet_bootstrap.cc:492] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19: No bootstrap required, opened a new log
I20260925 02:43:45.727459 6991 ts_tablet_manager.cc:1397] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20260925 02:43:45.727596 6992 ts_tablet_manager.cc:1397] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20260925 02:43:45.728025 6993 raft_consensus.cc:348] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.728367 6993 raft_consensus.cc:374] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:45.728338 6992 raft_consensus.cc:348] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.728484 6993 raft_consensus.cc:729] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4290762e134c4da9871393d120cb6e, State: Initialized, Role: FOLLOWER
I20260925 02:43:45.728399 6991 raft_consensus.cc:348] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.728614 6992 raft_consensus.cc:374] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:45.728688 6992 raft_consensus.cc:729] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8851b3d1d3b241838d3157f97317db19, State: Initialized, Role: FOLLOWER
I20260925 02:43:45.728705 6991 raft_consensus.cc:374] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:45.728806 6991 raft_consensus.cc:729] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13d6d2c53d6b46fbb76e948a855d61e8, State: Initialized, Role: FOLLOWER
I20260925 02:43:45.728965 6992 consensus_queue.cc:260] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.729445 6999 tablet_replica.cc:439] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } }
I20260925 02:43:45.729844 6999 tablet_replica.cc:442] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:45.730103 6993 consensus_queue.cc:260] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.730329 6992 ts_tablet_manager.cc:1428] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:43:45.729172 6991 consensus_queue.cc:260] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.730882 6998 tablet_replica.cc:439] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } }
I20260925 02:43:45.731238 6998 tablet_replica.cc:442] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:45.732317 6993 ts_tablet_manager.cc:1428] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260925 02:43:45.732211 6997 tablet_replica.cc:439] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } }
I20260925 02:43:45.732573 6997 tablet_replica.cc:442] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:45.733373 6991 ts_tablet_manager.cc:1428] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260925 02:43:45.958927 6998 raft_consensus.cc:482] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:43:45.959174 6998 raft_consensus.cc:504] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.959885 6998 leader_election.cc:288] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8851b3d1d3b241838d3157f97317db19 (127.5.56.129:34185), 8a4290762e134c4da9871393d120cb6e (127.5.56.131:43561)
I20260925 02:43:45.960419 6639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff49ff87b10430b8d6f604a352fd605" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8851b3d1d3b241838d3157f97317db19" is_pre_election: true
I20260925 02:43:45.960493 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff49ff87b10430b8d6f604a352fd605" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4290762e134c4da9871393d120cb6e" is_pre_election: true
I20260925 02:43:45.960731 6639 raft_consensus.cc:2454] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 0.
I20260925 02:43:45.960732 6920 raft_consensus.cc:2454] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 0.
I20260925 02:43:45.961193 6714 leader_election.cc:302] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13d6d2c53d6b46fbb76e948a855d61e8, 8851b3d1d3b241838d3157f97317db19; no voters:
I20260925 02:43:45.961449 6998 raft_consensus.cc:2781] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:43:45.961583 6998 raft_consensus.cc:482] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:43:45.961714 6998 raft_consensus.cc:3037] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:45.963884 6998 raft_consensus.cc:504] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.964452 6998 leader_election.cc:288] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 election: Requested vote from peers 8851b3d1d3b241838d3157f97317db19 (127.5.56.129:34185), 8a4290762e134c4da9871393d120cb6e (127.5.56.131:43561)
I20260925 02:43:45.964988 6920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff49ff87b10430b8d6f604a352fd605" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a4290762e134c4da9871393d120cb6e"
I20260925 02:43:45.964988 6639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff49ff87b10430b8d6f604a352fd605" candidate_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8851b3d1d3b241838d3157f97317db19"
I20260925 02:43:45.965236 6639 raft_consensus.cc:3037] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:45.965236 6920 raft_consensus.cc:3037] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:45.968151 6920 raft_consensus.cc:2454] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 1.
I20260925 02:43:45.968158 6639 raft_consensus.cc:2454] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13d6d2c53d6b46fbb76e948a855d61e8 in term 1.
I20260925 02:43:45.968622 6714 leader_election.cc:302] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13d6d2c53d6b46fbb76e948a855d61e8, 8851b3d1d3b241838d3157f97317db19, 8a4290762e134c4da9871393d120cb6e; no voters:
I20260925 02:43:45.968927 6998 raft_consensus.cc:2781] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:45.969129 6998 raft_consensus.cc:686] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 LEADER]: Becoming Leader. State: Replica: 13d6d2c53d6b46fbb76e948a855d61e8, State: Running, Role: LEADER
I20260925 02:43:45.969422 6998 consensus_queue.cc:237] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } }
I20260925 02:43:45.971280 7012 tablet_replica.cc:439] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: TxnStatusTablet state changed. Reason: New leader 13d6d2c53d6b46fbb76e948a855d61e8. Latest consensus state: current_term: 1 leader_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } }
I20260925 02:43:45.971563 7012 tablet_replica.cc:442] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:43:45.973137 7033 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:43:45.973380 7033 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:43:45.973606 6479 catalog_manager.cc:5495] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13d6d2c53d6b46fbb76e948a855d61e8 (127.5.56.130). New cstate: current_term: 1 leader_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } health_report { overall_health: UNKNOWN } } }
I20260925 02:43:46.006055 7012 consensus_queue.cc:1035] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:46.015877 7012 consensus_queue.cc:1035] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:46.053757 7027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:46.032753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:35521 (local address 127.0.0.1:36936)
0925 02:43:46.033311 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:46.033332 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:46.033497 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:46.044370 (+ 10873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:46.044376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:46.044393 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:43:46.044634 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:46.044646 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.046465 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.046471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:46.048134 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:46.048151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.052390 (+ 4239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.052397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:46.052599 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:46.052666 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:43:46.053127 (+ 461us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:43:46.053163 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:43:46.053272 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:43:46.053431 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375}
I20260925 02:43:46.053820 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:46.033782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36936 (local address 127.5.56.130:35521)
0925 02:43:46.043848 (+ 10066us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:46.043855 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:46.043885 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:43:46.043969 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:46.043975 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:46.044062 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:46.044783 (+ 721us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:43:46.045033 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.046245 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.050162 (+ 3917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.052213 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.052358 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:46.052882 (+ 524us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:43:46.053020 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:43:46.053122 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:46.053294 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 02:43:46.053376 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9882,"thread_start_us":10392,"threads_started":1}
I20260925 02:43:46.058239 6639 raft_consensus.cc:1261] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19 [term 1 FOLLOWER]: Refusing update from remote peer 13d6d2c53d6b46fbb76e948a855d61e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:46.058377 6920 raft_consensus.cc:1261] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e [term 1 FOLLOWER]: Refusing update from remote peer 13d6d2c53d6b46fbb76e948a855d61e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:46.059033 7012 consensus_queue.cc:1035] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:46.059612 6998 consensus_queue.cc:1035] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:46.063542 6997 tablet_replica.cc:439] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e: TxnStatusTablet state changed. Reason: New leader 13d6d2c53d6b46fbb76e948a855d61e8. Latest consensus state: current_term: 1 leader_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } }
I20260925 02:43:46.063833 6997 tablet_replica.cc:442] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:46.071514 7012 tablet_replica.cc:439] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } }
I20260925 02:43:46.071950 7012 tablet_replica.cc:442] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:43:46.078312 6997 tablet_replica.cc:439] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } }
I20260925 02:43:46.078604 6997 tablet_replica.cc:442] T fff49ff87b10430b8d6f604a352fd605 P 8a4290762e134c4da9871393d120cb6e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:46.085065 6999 tablet_replica.cc:439] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19: TxnStatusTablet state changed. Reason: New leader 13d6d2c53d6b46fbb76e948a855d61e8. Latest consensus state: current_term: 1 leader_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } }
I20260925 02:43:46.085393 6999 tablet_replica.cc:442] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:46.086376 6998 tablet_replica.cc:439] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } }
I20260925 02:43:46.086848 6998 tablet_replica.cc:442] T fff49ff87b10430b8d6f604a352fd605 P 13d6d2c53d6b46fbb76e948a855d61e8: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:43:46.091611 6999 tablet_replica.cc:439] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8851b3d1d3b241838d3157f97317db19" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34185 } } peers { permanent_uuid: "13d6d2c53d6b46fbb76e948a855d61e8" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35521 } } peers { permanent_uuid: "8a4290762e134c4da9871393d120cb6e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43561 } } }
I20260925 02:43:46.091917 6999 tablet_replica.cc:442] T fff49ff87b10430b8d6f604a352fd605 P 8851b3d1d3b241838d3157f97317db19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:46.104116 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:46.097417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:47655 (local address 127.5.56.130:35521)
0925 02:43:46.097585 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:46.097589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:46.098313 (+ 724us) server_negotiation.cc:407] Connection header received
0925 02:43:46.098548 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:46.098554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:46.098643 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:46.098752 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:43:46.099273 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.100433 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.102002 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.102692 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.102817 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:46.103247 (+ 430us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:43:46.103387 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:43:46.103482 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:46.103695 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 02:43:46.103782 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:43:46.104139 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:46.097419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:35521 (local address 127.5.56.130:47655)
0925 02:43:46.098195 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:46.098217 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:46.098352 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:46.098768 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:46.098771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:46.098781 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:43:46.099112 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:46.099122 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.100569 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.100577 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:46.101818 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:46.101830 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.102868 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.102872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:46.103013 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:46.103072 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:43:46.103548 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:43:46.103568 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:43:46.103646 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:43:46.103754 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":401,"threads_started":1}
I20260925 02:43:46.173835 6478 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:46.174137 6478 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:46.177428 6478 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30ba619501044e739ae9cbc5f46a6c68: Sending DeleteTablet for 3 replicas of tablet d0b0db2750a54370a1513470119121b8
I20260925 02:43:46.178318 6619 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b0db2750a54370a1513470119121b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:46 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:52014
I20260925 02:43:46.178426 6760 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b0db2750a54370a1513470119121b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:46 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:36926
I20260925 02:43:46.178704 6900 tablet_service.cc:1514] Processing DeleteTablet for tablet d0b0db2750a54370a1513470119121b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:46 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:60290
I20260925 02:43:46.188354 7055 tablet_replica.cc:330] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e: stopping tablet replica
I20260925 02:43:46.188530 7053 tablet_replica.cc:330] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19: stopping tablet replica
I20260925 02:43:46.188782 7053 raft_consensus.cc:2229] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:46.189718 7053 raft_consensus.cc:2258] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:46.189998 7055 raft_consensus.cc:2229] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:46.191009 7055 raft_consensus.cc:2258] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:46.191617 7053 ts_tablet_manager.cc:1905] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:46.192044 7054 tablet_replica.cc:330] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8: stopping tablet replica
I20260925 02:43:46.192281 7054 raft_consensus.cc:2229] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:43:46.192771 7054 raft_consensus.cc:2258] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:46.194031 7055 ts_tablet_manager.cc:1905] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:46.199527 7054 ts_tablet_manager.cc:1905] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:46.200763 7053 ts_tablet_manager.cc:1918] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:43:46.201092 7053 log.cc:1196] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/d0b0db2750a54370a1513470119121b8
I20260925 02:43:46.201898 7053 ts_tablet_manager.cc:1939] T d0b0db2750a54370a1513470119121b8 P 8851b3d1d3b241838d3157f97317db19: Deleting consensus metadata
I20260925 02:43:46.203716 6466 catalog_manager.cc:4841] TS 8851b3d1d3b241838d3157f97317db19 (127.5.56.129:34185): tablet d0b0db2750a54370a1513470119121b8 (table test-table [id=dd5b46db5d1649cfb479390341ec838a]) successfully deleted
I20260925 02:43:46.207003 7055 ts_tablet_manager.cc:1918] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:43:46.207145 7055 log.cc:1196] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/d0b0db2750a54370a1513470119121b8
I20260925 02:43:46.207684 7055 ts_tablet_manager.cc:1939] T d0b0db2750a54370a1513470119121b8 P 8a4290762e134c4da9871393d120cb6e: Deleting consensus metadata
I20260925 02:43:46.208930 6464 catalog_manager.cc:4841] TS 8a4290762e134c4da9871393d120cb6e (127.5.56.131:43561): tablet d0b0db2750a54370a1513470119121b8 (table test-table [id=dd5b46db5d1649cfb479390341ec838a]) successfully deleted
I20260925 02:43:46.210546 7054 ts_tablet_manager.cc:1918] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:43:46.210770 7054 log.cc:1196] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/d0b0db2750a54370a1513470119121b8
I20260925 02:43:46.211381 7054 ts_tablet_manager.cc:1939] T d0b0db2750a54370a1513470119121b8 P 13d6d2c53d6b46fbb76e948a855d61e8: Deleting consensus metadata
I20260925 02:43:46.212615 6465 catalog_manager.cc:4841] TS 13d6d2c53d6b46fbb76e948a855d61e8 (127.5.56.130:35521): tablet d0b0db2750a54370a1513470119121b8 (table test-table [id=dd5b46db5d1649cfb479390341ec838a]) successfully deleted
Sep 25 02:43:46 dist-test-slave-n563 krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:43:46.231690 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:46.191358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52030 (local address 127.5.56.129:34185)
0925 02:43:46.213425 (+ 22067us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:46.213430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:46.213446 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:43:46.213495 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:46.213498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:46.213565 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:46.213649 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:46.215650 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.216466 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.218357 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.219413 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.219600 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:46.224855 (+ 5255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:46.224873 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:46.224890 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:46.224929 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:46.228572 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:46.229285 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:46.229295 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:46.229299 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:46.229461 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:46.229830 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:46.229836 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:46.229839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:46.230341 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:46.230681 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:46.231012 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 02:43:46.231261 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21943,"thread_start_us":21834,"threads_started":1}
W20260925 02:43:46.232750 6685 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52030
I20260925 02:43:46.252671 7061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:46.236041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59530 (local address 127.5.56.190:46463)
0925 02:43:46.238791 (+ 2750us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:46.238796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:46.238813 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:43:46.238865 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:46.238868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:46.238930 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:46.239019 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:46.240681 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.241628 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.243512 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.243931 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.244033 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:46.246915 (+ 2882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:46.246920 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:46.246923 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:46.246971 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:46.249843 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:46.250699 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:46.250703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:46.250706 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:46.250831 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:46.251239 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:46.251243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:46.251245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:46.251502 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:46.251755 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:46.252126 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 02:43:46.252294 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2641,"thread_start_us":2521,"threads_started":1}
W20260925 02:43:46.253463 6478 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59530
I20260925 02:43:46.255411 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 6552
I20260925 02:43:46.270689 7057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:46.256635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.5.56.129:34185)
0925 02:43:46.256939 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:46.256946 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:46.256970 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:43:46.257126 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:46.257131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:46.257211 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:46.257315 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:46.258353 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.259467 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.261014 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.261674 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.261804 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:46.264585 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:46.264592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:46.264596 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:46.264637 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:46.267958 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:46.268739 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:46.268744 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:46.268748 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:46.268897 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:46.269269 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:46.269273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:46.269276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:46.269573 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:46.269822 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:46.270114 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 02:43:46.270325 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 02:43:46.271375 6685 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52040
W20260925 02:43:46.271972 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 6552: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:43:46.272153 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 6552
I20260925 02:43:46.298648 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 6694
Sep 25 02:43:46 dist-test-slave-n563 krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:43:46.315809 7040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:46.299526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36952 (local address 127.5.56.130:35521)
0925 02:43:46.299716 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:46.299724 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:46.299750 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:43:46.299939 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:46.299948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:46.300054 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:46.300171 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:46.301614 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.302853 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.304059 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.304735 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.304877 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:46.309316 (+ 4439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:46.309332 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:46.309348 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:46.309381 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:46.312782 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:46.313649 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:46.313659 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:46.313661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:46.313807 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:46.314270 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:46.314275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:46.314278 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:46.314703 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:46.314949 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:46.315300 (+ 351us) server_negotiation.cc:299] Negotiation successful
0925 02:43:46.315470 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260925 02:43:46.316538 6827 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36952
W20260925 02:43:46.317198 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 6694: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:43:46.317376 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 6694
I20260925 02:43:46.342187 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 6836
Sep 25 02:43:46 dist-test-slave-n563 krb5kdc[6434](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304225, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:43:46.357739 7063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:46.343010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.5.56.131:43561)
0925 02:43:46.343461 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:46.343465 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:46.343482 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:43:46.343540 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:46.343544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:46.343606 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:46.343698 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:46.344891 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.345584 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.346889 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:46.347338 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:46.347433 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:46.351586 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:46.351601 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:46.351618 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:46.351660 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:46.354922 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:46.355656 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:46.355663 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:46.355666 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:46.355795 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:46.356179 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:46.356184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:46.356188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:46.356561 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:46.356856 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:46.357156 (+ 300us) server_negotiation.cc:299] Negotiation successful
0925 02:43:46.357384 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":229,"threads_started":1}
W20260925 02:43:46.358544 6967 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60300
W20260925 02:43:46.359273 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 6836: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:43:46.359434 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 6836
I20260925 02:43:46.377663 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 6447
I20260925 02:43:46.378914 6520 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:43:46.494477 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 6447
2026-09-25T02:43:46Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4952 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:43:46 dist-test-slave-n563 krb5kdc[7067](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:43:46 dist-test-slave-n563 krb5kdc[7067](info): set up 2 sockets
Sep 25 02:43:46 dist-test-slave-n563 krb5kdc[7067](info): commencing operation
krb5kdc: starting...
W20260925 02:43:47.587867 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:43:47 dist-test-slave-n563 krb5kdc[7067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:43:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:43:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:43:47.785902 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:38925
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:38925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:48.113106 7080 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:48.113421 7080 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:48.113581 7080 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:48.121249 7080 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:43:48.121372 7080 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:43:48.121454 7080 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:48.121552 7080 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:48.121623 7080 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:43:48.121702 7080 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:43:48.132081 7080 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:38925
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:38925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:48.133441 7080 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:48.134871 7080 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:48.147089 7085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:48.147018 7088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:48.147042 7086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:48.147851 7080 server_base.cc:1034] running on GCE node
I20260925 02:43:48.148682 7080 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:48.150247 7080 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:48.151520 7080 hybrid_clock.cc:648] HybridClock initialized: now 1790304228151443 us; error 52 us; skew 500 ppm
I20260925 02:43:48.151880 7080 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:48 dist-test-slave-n563 krb5kdc[7067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304228, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:48.155227 7080 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:43:48.156780 7080 webserver.cc:460] Webserver started at http://127.5.56.190:37117/ using document root <none> and password file <none>
I20260925 02:43:48.157521 7080 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:48.157657 7080 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:48.157991 7080 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:48.161157 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "b18f1624cca745feb177d663df99ddab"
format_stamp: "Formatted at 2026-09-25 02:43:48 on dist-test-slave-n563"
server_key: "7f6fb6a2411715a9b96ce526eef37ac2"
server_key_iv: "bc4cbe14606e0ed3557f90a12c88dfe2"
server_key_version: "encryptionkey@0"
I20260925 02:43:48.161999 7080 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "b18f1624cca745feb177d663df99ddab"
format_stamp: "Formatted at 2026-09-25 02:43:48 on dist-test-slave-n563"
server_key: "7f6fb6a2411715a9b96ce526eef37ac2"
server_key_iv: "bc4cbe14606e0ed3557f90a12c88dfe2"
server_key_version: "encryptionkey@0"
I20260925 02:43:48.167542 7080 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 02:43:48.171205 7095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:48.172540 7080 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:43:48.172737 7080 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "b18f1624cca745feb177d663df99ddab"
format_stamp: "Formatted at 2026-09-25 02:43:48 on dist-test-slave-n563"
server_key: "7f6fb6a2411715a9b96ce526eef37ac2"
server_key_iv: "bc4cbe14606e0ed3557f90a12c88dfe2"
server_key_version: "encryptionkey@0"
I20260925 02:43:48.172969 7080 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:48.183952 7080 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:48.188102 7080 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:48.188383 7080 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:48.209945 7080 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:38925
I20260925 02:43:48.209971 7157 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:38925 every 8 connection(s)
I20260925 02:43:48.211417 7080 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:43:48.215112 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 7080
I20260925 02:43:48.215342 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:43:48.215615 7158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:48.216188 5346 external_mini_cluster.cc:1428] Setting key 55459c886b3d3f839346cf0cc4d950e8
I20260925 02:43:48.230798 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab: Bootstrap starting.
Sep 25 02:43:48 dist-test-slave-n563 krb5kdc[7067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:48.234768 7158 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:48.236199 7158 log.cc:824] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:48.239377 7158 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab: No bootstrap required, opened a new log
I20260925 02:43:48.241145 7161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:48.218272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60280 (local address 127.5.56.190:38925)
0925 02:43:48.219147 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:48.219159 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:48.219204 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:43:48.220840 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:48.220866 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:48.221296 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:48.221631 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:48.223511 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:48.224860 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:48.226741 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:48.228039 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:48.228283 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:48.233116 (+ 4833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:48.233125 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:48.233141 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:48.233188 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:48.237016 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:48.237785 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:48.237792 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:48.237799 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:48.237958 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:48.238360 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:48.238370 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:48.238372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:48.238941 (+ 569us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:48.239263 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:48.239693 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 02:43:48.240137 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":430,"threads_started":1}
I20260925 02:43:48.247808 7158 raft_consensus.cc:348] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18f1624cca745feb177d663df99ddab" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38925 } }
I20260925 02:43:48.248273 7158 raft_consensus.cc:374] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:48.248389 7158 raft_consensus.cc:729] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18f1624cca745feb177d663df99ddab, State: Initialized, Role: FOLLOWER
I20260925 02:43:48.248950 7158 consensus_queue.cc:260] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18f1624cca745feb177d663df99ddab" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38925 } }
I20260925 02:43:48.249185 7158 raft_consensus.cc:388] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:43:48.249322 7158 raft_consensus.cc:482] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:43:48.249493 7158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:48.251813 7158 raft_consensus.cc:504] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18f1624cca745feb177d663df99ddab" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38925 } }
I20260925 02:43:48.252342 7158 leader_election.cc:302] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b18f1624cca745feb177d663df99ddab; no voters:
I20260925 02:43:48.253019 7158 leader_election.cc:288] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:43:48.253177 7163 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:48.253880 7163 raft_consensus.cc:686] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [term 1 LEADER]: Becoming Leader. State: Replica: b18f1624cca745feb177d663df99ddab, State: Running, Role: LEADER
I20260925 02:43:48.254987 7163 consensus_queue.cc:237] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18f1624cca745feb177d663df99ddab" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38925 } }
I20260925 02:43:48.255936 7158 sys_catalog.cc:564] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:43:48.258090 7165 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [sys.catalog]: SysCatalogTable state changed. Reason: New leader b18f1624cca745feb177d663df99ddab. Latest consensus state: current_term: 1 leader_uuid: "b18f1624cca745feb177d663df99ddab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18f1624cca745feb177d663df99ddab" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38925 } } }
I20260925 02:43:48.258538 7165 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [sys.catalog]: This master's current role is: LEADER
I20260925 02:43:48.258230 7164 sys_catalog.cc:455] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b18f1624cca745feb177d663df99ddab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b18f1624cca745feb177d663df99ddab" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38925 } } }
I20260925 02:43:48.259568 7164 sys_catalog.cc:458] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab [sys.catalog]: This master's current role is: LEADER
I20260925 02:43:48.261327 7168 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:43:48.266657 7168 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:43:48.277068 7168 catalog_manager.cc:1347] Generated new cluster ID: 2de59f5b09d94cdda8e581234a05d145
I20260925 02:43:48.277200 7168 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:43:48.290616 7168 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:43:48.291958 7168 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:43:48.300411 7168 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab: Generated new TSK 0
I20260925 02:43:48.301563 7168 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:43:48.356169 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38925
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:48.693138 7184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:48.693506 7184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:48.693682 7184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:48.700980 7184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:48.701155 7184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:48.701318 7184 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:43:48.701408 7184 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:43:48.701560 7184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:43:48.711618 7184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.56.190:38925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:48.713069 7184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:48.714514 7184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:48.728075 7192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:48.728077 7190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:48.728091 7189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:48.728976 7184 server_base.cc:1034] running on GCE node
I20260925 02:43:48.729643 7184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:48.730708 7184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:48.732002 7184 hybrid_clock.cc:648] HybridClock initialized: now 1790304228731938 us; error 50 us; skew 500 ppm
I20260925 02:43:48.732340 7184 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:48 dist-test-slave-n563 krb5kdc[7067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304228, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:48.735672 7184 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:43:48.737291 7184 webserver.cc:460] Webserver started at http://127.5.56.129:42063/ using document root <none> and password file <none>
I20260925 02:43:48.738154 7184 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:48.738303 7184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:48.738662 7184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:48.741724 7184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "b42375594c7342c4a0e410b82148801b"
format_stamp: "Formatted at 2026-09-25 02:43:48 on dist-test-slave-n563"
server_key: "7bb9329956546811a0333ee9d4582330"
server_key_iv: "a1442bfdfd7b08a798a9634c43531ea9"
server_key_version: "encryptionkey@0"
I20260925 02:43:48.742596 7184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "b42375594c7342c4a0e410b82148801b"
format_stamp: "Formatted at 2026-09-25 02:43:48 on dist-test-slave-n563"
server_key: "7bb9329956546811a0333ee9d4582330"
server_key_iv: "a1442bfdfd7b08a798a9634c43531ea9"
server_key_version: "encryptionkey@0"
I20260925 02:43:48.747995 7184 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:43:48.751840 7199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:48.753315 7184 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:43:48.753506 7184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "b42375594c7342c4a0e410b82148801b"
format_stamp: "Formatted at 2026-09-25 02:43:48 on dist-test-slave-n563"
server_key: "7bb9329956546811a0333ee9d4582330"
server_key_iv: "a1442bfdfd7b08a798a9634c43531ea9"
server_key_version: "encryptionkey@0"
I20260925 02:43:48.753726 7184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:48.771636 7184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:48.776183 7184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:48.776530 7184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:48.779829 7184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:48.779985 7184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:48.780119 7184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:48.780205 7184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:43:48 dist-test-slave-n563 krb5kdc[7067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304228, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:48.808132 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:48.780257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:54335 (local address 127.5.56.190:38925)
0925 02:43:48.781168 (+ 911us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:48.781173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:48.781954 (+ 781us) server_negotiation.cc:407] Connection header received
0925 02:43:48.784224 (+ 2270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:48.784230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:48.784341 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:48.784509 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:48.787024 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:48.788131 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:48.789790 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:48.790200 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:48.790335 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:48.800084 (+ 9749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:48.800099 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:48.800107 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:48.800170 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:48.803960 (+ 3790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:48.805053 (+ 1093us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:48.805060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:48.805063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:48.805284 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:48.805949 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:48.805952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:48.805954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:48.806250 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:48.806366 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:48.807663 (+ 1297us) server_negotiation.cc:299] Negotiation successful
0925 02:43:48.807859 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":695,"threads_started":1}
I20260925 02:43:48.810808 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:48.780937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38925 (local address 127.5.56.129:54335)
0925 02:43:48.781793 (+ 856us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:48.781832 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:48.783847 (+ 2015us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:48.784811 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:48.784822 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:48.785683 (+ 861us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:48.786765 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:48.786797 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:48.788286 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:48.788292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:48.789643 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:48.789654 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:48.790465 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:48.790476 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:48.790769 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:48.796355 (+ 5586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:48.796412 (+ 57us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:48.799701 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:48.804168 (+ 4467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:48.804180 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:48.804215 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:48.804906 (+ 691us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:48.805434 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:48.805442 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:48.805449 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:48.805805 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:48.806408 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:48.806420 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:48.807094 (+ 674us) client_negotiation.cc:769] Sending connection context
0925 02:43:48.807552 (+ 458us) client_negotiation.cc:240] Negotiation successful
0925 02:43:48.808681 (+ 1129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":363,"threads_started":1}
I20260925 02:43:48.833307 7274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:48.816855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38925 (local address 127.5.56.129:41691)
0925 02:43:48.817652 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:48.817678 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:48.817869 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:48.818236 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:48.818244 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:48.819064 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:48.819646 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:48.819668 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:48.821038 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:48.821046 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:48.822795 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:48.822815 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:48.824057 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:48.824069 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:48.824223 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:48.824858 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:48.824876 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:48.825591 (+ 715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:48.829792 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:48.829799 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:48.829806 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:48.830442 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:48.830928 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:48.830935 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:48.830941 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:48.831140 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:48.831808 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:48.831812 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:48.832089 (+ 277us) client_negotiation.cc:769] Sending connection context
0925 02:43:48.832284 (+ 195us) client_negotiation.cc:240] Negotiation successful
0925 02:43:48.832650 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":394,"threads_started":1}
I20260925 02:43:48.833900 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:48.816902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:41691 (local address 127.5.56.190:38925)
0925 02:43:48.817153 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:48.817163 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:48.817777 (+ 614us) server_negotiation.cc:407] Connection header received
0925 02:43:48.817995 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:48.817999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:48.818083 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:48.818171 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:48.819838 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:48.820829 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:48.822971 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:48.823828 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:48.823982 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:48.825800 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:48.825810 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:48.825817 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:48.825865 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:48.829631 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:48.830621 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:48.830625 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:48.830627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:48.830757 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:48.831269 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:48.831274 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:48.831278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:48.831684 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:48.831834 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:48.833405 (+ 1571us) server_negotiation.cc:299] Negotiation successful
0925 02:43:48.833682 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260925 02:43:48.839273 7184 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:36009
I20260925 02:43:48.839313 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:36009 every 8 connection(s)
I20260925 02:43:48.840503 7184 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:43:48.842519 7321 heartbeater.cc:347] Connected to a master server at 127.5.56.190:38925
I20260925 02:43:48.842720 7321 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:48.843323 7321 heartbeater.cc:511] Master 127.5.56.190:38925 requested a full tablet report, sending...
I20260925 02:43:48.845247 7113 ts_manager.cc:194] Registered new tserver with Master: b42375594c7342c4a0e410b82148801b (127.5.56.129:36009)
I20260925 02:43:48.846764 7113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:54335
I20260925 02:43:48.848325 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 7184
I20260925 02:43:48.848511 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:43:48.849123 5346 external_mini_cluster.cc:1428] Setting key 519318b37c7e423b8a1914c3fe72091a
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:43:48.891719 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38925
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:49.220371 7326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:49.220640 7326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:49.220820 7326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:49.230207 7326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:49.230363 7326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:49.230535 7326 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:43:49.230624 7326 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:43:49.230731 7326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:43:49.242424 7326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.56.190:38925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:49.243830 7326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:49.245217 7326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:49.259624 7331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:49.259603 7332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:49.259912 7334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:49.260468 7326 server_base.cc:1034] running on GCE node
I20260925 02:43:49.261088 7326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:49.262034 7326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:49.263288 7326 hybrid_clock.cc:648] HybridClock initialized: now 1790304229263226 us; error 46 us; skew 500 ppm
I20260925 02:43:49.263655 7326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:49 dist-test-slave-n563 krb5kdc[7067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304229, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:49.266853 7326 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:43:49.268437 7326 webserver.cc:460] Webserver started at http://127.5.56.130:41275/ using document root <none> and password file <none>
I20260925 02:43:49.269187 7326 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:49.269315 7326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:49.269671 7326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:49.272782 7326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "86438781b1934792b0d1f9e24153bab0"
format_stamp: "Formatted at 2026-09-25 02:43:49 on dist-test-slave-n563"
server_key: "d577369ce93944791ccef3698467f63b"
server_key_iv: "831ffd2df4d90307c9d7154168595153"
server_key_version: "encryptionkey@0"
I20260925 02:43:49.273578 7326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "86438781b1934792b0d1f9e24153bab0"
format_stamp: "Formatted at 2026-09-25 02:43:49 on dist-test-slave-n563"
server_key: "d577369ce93944791ccef3698467f63b"
server_key_iv: "831ffd2df4d90307c9d7154168595153"
server_key_version: "encryptionkey@0"
I20260925 02:43:49.279184 7326 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:43:49.282924 7341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:49.284421 7326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 02:43:49.284610 7326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "86438781b1934792b0d1f9e24153bab0"
format_stamp: "Formatted at 2026-09-25 02:43:49 on dist-test-slave-n563"
server_key: "d577369ce93944791ccef3698467f63b"
server_key_iv: "831ffd2df4d90307c9d7154168595153"
server_key_version: "encryptionkey@0"
I20260925 02:43:49.284844 7326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:49.303929 7326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:49.308349 7326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:49.308665 7326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:49.311676 7326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:49.311770 7326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:49.311954 7326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:49.312062 7326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:43:49 dist-test-slave-n563 krb5kdc[7067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304229, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:49.333825 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:49.312164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:42409 (local address 127.5.56.190:38925)
0925 02:43:49.312379 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:49.312388 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:49.313997 (+ 1609us) server_negotiation.cc:407] Connection header received
0925 02:43:49.316157 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:49.316164 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:49.316260 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:49.316395 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:49.318916 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.320015 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.321674 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.322130 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.322258 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:49.327623 (+ 5365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:49.327629 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:49.327633 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:49.327667 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:49.330615 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:49.331401 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:49.331412 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:49.331414 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:49.331533 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:49.331995 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:49.331999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:49.332002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:49.332301 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:49.332468 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:49.333238 (+ 770us) server_negotiation.cc:299] Negotiation successful
0925 02:43:49.333470 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:43:49.335237 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:49.312854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38925 (local address 127.5.56.130:42409)
0925 02:43:49.313809 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:49.313863 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:49.315838 (+ 1975us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:49.316852 (+ 1014us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:49.316863 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:49.317689 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:49.318717 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:49.318752 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.320174 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.320180 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:49.321532 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:49.321544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.322377 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.322407 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:49.322639 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:49.323629 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:49.323674 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:49.327354 (+ 3680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:49.330790 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:49.330799 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:49.330813 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:49.331266 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:49.331667 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:49.331672 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:49.331674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:49.331867 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:49.332431 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:49.332441 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:49.332831 (+ 390us) client_negotiation.cc:769] Sending connection context
0925 02:43:49.333146 (+ 315us) client_negotiation.cc:240] Negotiation successful
0925 02:43:49.333730 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":422,"threads_started":1}
I20260925 02:43:49.358999 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:49.343609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:51923 (local address 127.5.56.190:38925)
0925 02:43:49.343858 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:49.343868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:49.344454 (+ 586us) server_negotiation.cc:407] Connection header received
0925 02:43:49.344646 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:49.344656 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:49.344765 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:49.344922 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:49.345839 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.347287 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.348613 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.349515 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.349700 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:49.352113 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:49.352123 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:49.352128 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:49.352179 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:49.355752 (+ 3573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:49.356716 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:49.356721 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:49.356725 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:49.356868 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:49.357278 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:49.357284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:49.357287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:49.357745 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:49.357899 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:49.358412 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 02:43:49.358648 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:43:49.359109 7404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:49.343613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38925 (local address 127.5.56.130:51923)
0925 02:43:49.344328 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:49.344353 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:49.344512 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:49.344928 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:49.344931 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:49.345368 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:49.345683 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:49.345692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.347452 (+ 1760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.347461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:49.348459 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:49.348472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.349723 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.349728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:49.349872 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:49.350680 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:49.350712 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:49.351863 (+ 1151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:49.355932 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:49.355943 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:49.355948 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:49.356524 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:49.357020 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:49.357028 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:49.357031 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:49.357169 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:49.357872 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:49.357877 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:49.358127 (+ 250us) client_negotiation.cc:769] Sending connection context
0925 02:43:49.358284 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 02:43:49.358648 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":377,"threads_started":1}
I20260925 02:43:49.374025 7326 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:39747
I20260925 02:43:49.374063 7461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:39747 every 8 connection(s)
I20260925 02:43:49.375494 7326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:43:49.377823 7462 heartbeater.cc:347] Connected to a master server at 127.5.56.190:38925
I20260925 02:43:49.378055 7462 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:49.378787 7462 heartbeater.cc:511] Master 127.5.56.190:38925 requested a full tablet report, sending...
I20260925 02:43:49.380200 7113 ts_manager.cc:194] Registered new tserver with Master: 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747)
I20260925 02:43:49.381166 7113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:42409
I20260925 02:43:49.381264 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 7326
I20260925 02:43:49.381434 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:43:49.382051 5346 external_mini_cluster.cc:1428] Setting key ff5d1cb6c3136e5336e4d943ae4ddc11
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:43:49.423861 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38925
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:49.755187 7466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:49.755541 7466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:49.755712 7466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:49.763625 7466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:49.763773 7466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:49.763931 7466 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 02:43:49.764015 7466 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:43:49.764132 7466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:43:49.774081 7466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.56.190:38925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:49.775565 7466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:49.777037 7466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:49.790616 7471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:49.790616 7475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:49.791141 7472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:49.791538 7466 server_base.cc:1034] running on GCE node
I20260925 02:43:49.792209 7466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:49.793185 7466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:49.794452 7466 hybrid_clock.cc:648] HybridClock initialized: now 1790304229794347 us; error 59 us; skew 500 ppm
I20260925 02:43:49.794816 7466 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:49 dist-test-slave-n563 krb5kdc[7067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304229, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:49.798017 7466 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:43:49.799604 7466 webserver.cc:460] Webserver started at http://127.5.56.131:43317/ using document root <none> and password file <none>
I20260925 02:43:49.800251 7466 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:49.800354 7466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:49.800612 7466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:49.803582 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "a5909a1b97ed428d89e8d229b82dbe9e"
format_stamp: "Formatted at 2026-09-25 02:43:49 on dist-test-slave-n563"
server_key: "db54f5d2908bd167df8332c111677dcd"
server_key_iv: "c3c3e332ea9b8dd1f817f4711b106d96"
server_key_version: "encryptionkey@0"
I20260925 02:43:49.804291 7466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "a5909a1b97ed428d89e8d229b82dbe9e"
format_stamp: "Formatted at 2026-09-25 02:43:49 on dist-test-slave-n563"
server_key: "db54f5d2908bd167df8332c111677dcd"
server_key_iv: "c3c3e332ea9b8dd1f817f4711b106d96"
server_key_version: "encryptionkey@0"
I20260925 02:43:49.809410 7466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:43:49.812978 7482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:49.814352 7466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:43:49.814567 7466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "a5909a1b97ed428d89e8d229b82dbe9e"
format_stamp: "Formatted at 2026-09-25 02:43:49 on dist-test-slave-n563"
server_key: "db54f5d2908bd167df8332c111677dcd"
server_key_iv: "c3c3e332ea9b8dd1f817f4711b106d96"
server_key_version: "encryptionkey@0"
I20260925 02:43:49.814762 7466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:49.836560 7466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:49.842204 7466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:49.842649 7466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:49.845654 7466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:49.845803 7466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:49.845937 7466 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:49.846027 7466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:49.849526 7321 heartbeater.cc:503] Master 127.5.56.190:38925 was elected leader, sending a full tablet report...
Sep 25 02:43:49 dist-test-slave-n563 krb5kdc[7067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304229, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:49.873273 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:49.846611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:57699 (local address 127.5.56.190:38925)
0925 02:43:49.846810 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:49.846816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:49.848323 (+ 1507us) server_negotiation.cc:407] Connection header received
0925 02:43:49.851594 (+ 3271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:49.851600 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:49.851708 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:49.851782 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:49.854998 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.856044 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.859029 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.859679 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.859827 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:49.864972 (+ 5145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:49.864981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:49.864988 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:49.865047 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:49.869573 (+ 4526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:49.870753 (+ 1180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:49.870759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:49.870763 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:49.870904 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:49.871355 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:49.871360 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:49.871362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:49.871650 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:49.871824 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:49.872673 (+ 849us) server_negotiation.cc:299] Negotiation successful
0925 02:43:49.872909 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:43:49.874307 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:49.847155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38925 (local address 127.5.56.131:57699)
0925 02:43:49.848109 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:49.848164 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:49.851183 (+ 3019us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:49.852460 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:49.852478 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:49.853504 (+ 1026us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:49.854708 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:49.854747 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.856260 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.856268 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:49.858767 (+ 2499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:49.858789 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.859971 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.859980 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:49.860250 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:49.861095 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:49.861159 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:49.864620 (+ 3461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:49.869784 (+ 5164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:49.869792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:49.869807 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:49.870600 (+ 793us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:49.871021 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:49.871026 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:49.871030 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:49.871227 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:49.871786 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:49.871795 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:49.872228 (+ 433us) client_negotiation.cc:769] Sending connection context
0925 02:43:49.872514 (+ 286us) client_negotiation.cc:240] Negotiation successful
0925 02:43:49.873111 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":383,"threads_started":1}
I20260925 02:43:49.898998 7559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:49.881527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38925 (local address 127.5.56.131:57913)
0925 02:43:49.882978 (+ 1451us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:49.883008 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:49.883194 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:49.883619 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:49.883626 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:49.884414 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:49.884900 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:49.884926 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.885956 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.885972 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:49.888184 (+ 2212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:49.888203 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.889252 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.889259 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:49.889415 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:49.890311 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:49.890341 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:49.891192 (+ 851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:49.896011 (+ 4819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:49.896015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:49.896018 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:49.896464 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:49.896956 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:49.896963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:49.896968 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:49.897206 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:49.897800 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:49.897805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:49.898077 (+ 272us) client_negotiation.cc:769] Sending connection context
0925 02:43:49.898260 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 02:43:49.898652 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"thread_start_us":897,"threads_started":1}
I20260925 02:43:49.899127 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:49.881529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:57913 (local address 127.5.56.190:38925)
0925 02:43:49.881756 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:49.881765 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:49.883113 (+ 1348us) server_negotiation.cc:407] Connection header received
0925 02:43:49.883382 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:49.883387 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:49.883462 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:49.883558 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:49.885100 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.885801 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.888371 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.889009 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.889148 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:49.891417 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:49.891428 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:49.891436 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:49.891489 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:49.895831 (+ 4342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:49.896602 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:49.896611 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:49.896617 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:49.896812 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:49.897381 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:49.897389 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:49.897393 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:49.897677 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:49.897835 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:49.898456 (+ 621us) server_negotiation.cc:299] Negotiation successful
0925 02:43:49.898726 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:43:49.911774 7466 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:45591
I20260925 02:43:49.911777 7601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:45591 every 8 connection(s)
I20260925 02:43:49.913172 7466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:43:49.913779 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 7466
I20260925 02:43:49.913978 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:43:49.914492 5346 external_mini_cluster.cc:1428] Setting key f17edff8baa1fb4df5a918eb3b4d57e7
I20260925 02:43:49.915707 7602 heartbeater.cc:347] Connected to a master server at 127.5.56.190:38925
I20260925 02:43:49.915946 7602 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:49.916707 7602 heartbeater.cc:511] Master 127.5.56.190:38925 requested a full tablet report, sending...
I20260925 02:43:49.918169 7112 ts_manager.cc:194] Registered new tserver with Master: a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131:45591)
I20260925 02:43:49.919211 7112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:57699
I20260925 02:43:49.930282 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:43:49 dist-test-slave-n563 krb5kdc[7067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:43:49 dist-test-slave-n563 krb5kdc[7067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304229, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:49.979262 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:49.965151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60292 (local address 127.5.56.190:38925)
0925 02:43:49.965414 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:49.965422 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:49.965789 (+ 367us) server_negotiation.cc:407] Connection header received
0925 02:43:49.966007 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:49.966011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:49.966087 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:49.966194 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:49.967658 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.968447 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.969654 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:49.970108 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:49.970242 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:49.974077 (+ 3835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:49.974083 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:49.974086 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:49.974135 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:49.976731 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:49.977505 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:49.977512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:49.977516 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:49.977654 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:49.978012 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:49.978015 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:49.978017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:49.978282 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:49.978542 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:49.978775 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:43:49.978982 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260925 02:43:49.983827 7112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:43:49.991047 7112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:43:50.013123 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.002973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35408 (local address 127.5.56.130:39747)
0925 02:43:50.005545 (+ 2572us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.005554 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.005584 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:43:50.005683 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.005692 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.005967 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.006695 (+ 728us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:50.007177 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.008673 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.010975 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.011858 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.012371 (+ 513us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.012661 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.012758 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.012904 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2392,"thread_start_us":2263,"threads_started":1}
I20260925 02:43:50.013224 7615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.002795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39747 (local address 127.0.0.1:35408)
0925 02:43:50.003795 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.003851 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.004014 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.006224 (+ 2210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.006234 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.006264 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:50.006999 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.007016 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.008834 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.008840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.010817 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.010833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.012091 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.012097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.012361 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.012433 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.012567 (+ 134us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.012809 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":114,"thread_start_us":1076,"threads_started":1}
I20260925 02:43:50.015261 7620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.002966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45591 (local address 127.0.0.1:50958)
0925 02:43:50.005383 (+ 2417us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.005408 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.005549 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.006255 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.006259 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.006274 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:50.006821 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.006835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.009189 (+ 2354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.009196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.011756 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.011768 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.014423 (+ 2655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.014432 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.014593 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.014633 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.014731 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.014860 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2293,"mutex_wait_us":151,"thread_start_us":2965,"threads_started":1}
I20260925 02:43:50.015327 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.003556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50958 (local address 127.5.56.131:45591)
0925 02:43:50.005099 (+ 1543us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.005107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.005678 (+ 571us) server_negotiation.cc:407] Connection header received
0925 02:43:50.005782 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.005789 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.006069 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.006257 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:50.007068 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.009060 (+ 1992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.011953 (+ 2893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.013415 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.013555 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.013781 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.014792 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.014966 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1355,"thread_start_us":1154,"threads_started":1}
I20260925 02:43:50.016348 7617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.004491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36009 (local address 127.0.0.1:53172)
0925 02:43:50.006192 (+ 1701us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.006209 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.006314 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.007186 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.007190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.007217 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:50.007477 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.007484 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.009100 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.009111 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.013706 (+ 4595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.013725 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.015535 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.015558 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.015739 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.015772 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.015859 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.015978 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1566,"mutex_wait_us":171,"thread_start_us":773,"threads_started":1}
I20260925 02:43:50.016932 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.005096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.5.56.129:36009)
0925 02:43:50.005787 (+ 691us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.005795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.006487 (+ 692us) server_negotiation.cc:407] Connection header received
0925 02:43:50.006617 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.006643 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.006868 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.007049 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:50.007634 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.008963 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.013876 (+ 4913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.015030 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.015187 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.015380 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.016596 (+ 1216us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.016735 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":664,"threads_started":1}
I20260925 02:43:50.017223 7395 tablet_service.cc:1467] Processing CreateTablet for tablet 441ea6ef7c294a13a9564a8905f81e9f (DEFAULT_TABLE table=test-table [id=21211a5182a44a79810987517b6c9e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:50.018872 7395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441ea6ef7c294a13a9564a8905f81e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:50.019066 7532 tablet_service.cc:1467] Processing CreateTablet for tablet 441ea6ef7c294a13a9564a8905f81e9f (DEFAULT_TABLE table=test-table [id=21211a5182a44a79810987517b6c9e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:50.020087 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 441ea6ef7c294a13a9564a8905f81e9f (DEFAULT_TABLE table=test-table [id=21211a5182a44a79810987517b6c9e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:50.020435 7532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441ea6ef7c294a13a9564a8905f81e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:50.021063 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 441ea6ef7c294a13a9564a8905f81e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:50.031872 7622 tablet_bootstrap.cc:492] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: Bootstrap starting.
I20260925 02:43:50.033988 7623 tablet_bootstrap.cc:492] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: Bootstrap starting.
I20260925 02:43:50.035672 7624 tablet_bootstrap.cc:492] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: Bootstrap starting.
I20260925 02:43:50.037318 7622 tablet_bootstrap.cc:654] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:50.037315 7623 tablet_bootstrap.cc:654] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:50.038643 7624 tablet_bootstrap.cc:654] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:50.038692 7623 log.cc:824] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:50.038681 7622 log.cc:824] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:50.039876 7624 log.cc:824] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:50.042003 7622 tablet_bootstrap.cc:492] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: No bootstrap required, opened a new log
I20260925 02:43:50.042306 7623 tablet_bootstrap.cc:492] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: No bootstrap required, opened a new log
I20260925 02:43:50.042335 7622 ts_tablet_manager.cc:1397] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260925 02:43:50.042590 7624 tablet_bootstrap.cc:492] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: No bootstrap required, opened a new log
I20260925 02:43:50.042611 7623 ts_tablet_manager.cc:1397] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260925 02:43:50.042840 7624 ts_tablet_manager.cc:1397] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260925 02:43:50.049665 7624 raft_consensus.cc:348] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } }
I20260925 02:43:50.050153 7624 raft_consensus.cc:374] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:50.050261 7624 raft_consensus.cc:729] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5909a1b97ed428d89e8d229b82dbe9e, State: Initialized, Role: FOLLOWER
I20260925 02:43:50.051100 7624 consensus_queue.cc:260] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } }
I20260925 02:43:50.052265 7602 heartbeater.cc:503] Master 127.5.56.190:38925 was elected leader, sending a full tablet report...
I20260925 02:43:50.052995 7624 ts_tablet_manager.cc:1428] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260925 02:43:50.053310 7623 raft_consensus.cc:348] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } }
I20260925 02:43:50.053308 7622 raft_consensus.cc:348] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } }
I20260925 02:43:50.053754 7622 raft_consensus.cc:374] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:50.053754 7623 raft_consensus.cc:374] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:50.053859 7623 raft_consensus.cc:729] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86438781b1934792b0d1f9e24153bab0, State: Initialized, Role: FOLLOWER
I20260925 02:43:50.053858 7622 raft_consensus.cc:729] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42375594c7342c4a0e410b82148801b, State: Initialized, Role: FOLLOWER
I20260925 02:43:50.054488 7622 consensus_queue.cc:260] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } }
I20260925 02:43:50.054492 7623 consensus_queue.cc:260] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } }
I20260925 02:43:50.056365 7622 ts_tablet_manager.cc:1428] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20260925 02:43:50.056663 7623 ts_tablet_manager.cc:1428] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 02:43:50.056818 7462 heartbeater.cc:503] Master 127.5.56.190:38925 was elected leader, sending a full tablet report...
W20260925 02:43:50.092841 7322 tablet.cc:2369] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:43:50.126194 7631 raft_consensus.cc:482] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:43:50.126550 7631 raft_consensus.cc:504] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } }
W20260925 02:43:50.127796 7463 tablet.cc:2369] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:43:50.128350 7631 leader_election.cc:288] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747), a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131:45591)
I20260925 02:43:50.135293 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.128915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:41385 (local address 127.5.56.130:39747)
0925 02:43:50.129171 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.129180 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.129970 (+ 790us) server_negotiation.cc:407] Connection header received
0925 02:43:50.130167 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.130171 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.130229 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.130328 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:50.131010 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.131687 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.133467 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.134262 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.134479 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.134558 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.134884 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.135016 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:43:50.135531 7633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.128674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45591 (local address 127.5.56.129:45479)
0925 02:43:50.129792 (+ 1118us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.129808 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.129983 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.130301 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.130305 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.130329 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:50.130800 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.130809 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.131768 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.131775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.133341 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.133352 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.134714 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.134720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.134885 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.134914 (+ 29us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.135009 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.135130 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":979,"mutex_wait_us":21,"thread_start_us":915,"threads_started":1}
I20260925 02:43:50.135675 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.128963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:45479 (local address 127.5.56.131:45591)
0925 02:43:50.129215 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.129224 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.129907 (+ 683us) server_negotiation.cc:407] Connection header received
0925 02:43:50.130129 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.130133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.130191 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.130287 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:50.130944 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.131643 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.133490 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.134316 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.134517 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.134615 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.135048 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.135189 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:43:50.136078 7632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.128725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39747 (local address 127.5.56.129:41385)
0925 02:43:50.129853 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.129868 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.130019 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.130338 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.130343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.130365 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:50.130838 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.130846 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.131829 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.131834 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.133316 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.133332 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.134505 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.134512 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.134708 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.134741 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.134838 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.135713 (+ 875us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"mutex_wait_us":33,"thread_start_us":645,"threads_started":1}
I20260925 02:43:50.136682 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441ea6ef7c294a13a9564a8905f81e9f" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" is_pre_election: true
I20260925 02:43:50.137050 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441ea6ef7c294a13a9564a8905f81e9f" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86438781b1934792b0d1f9e24153bab0" is_pre_election: true
I20260925 02:43:50.137187 7552 raft_consensus.cc:2454] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 0.
I20260925 02:43:50.137518 7417 raft_consensus.cc:2454] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 0.
I20260925 02:43:50.137928 7202 leader_election.cc:302] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5909a1b97ed428d89e8d229b82dbe9e, b42375594c7342c4a0e410b82148801b; no voters:
I20260925 02:43:50.138415 7631 raft_consensus.cc:2781] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:43:50.138566 7631 raft_consensus.cc:482] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:43:50.138695 7631 raft_consensus.cc:3037] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:50.140919 7631 raft_consensus.cc:504] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } }
I20260925 02:43:50.141499 7631 leader_election.cc:288] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 election: Requested vote from peers 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747), a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131:45591)
I20260925 02:43:50.141880 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441ea6ef7c294a13a9564a8905f81e9f" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86438781b1934792b0d1f9e24153bab0"
I20260925 02:43:50.141928 7552 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "441ea6ef7c294a13a9564a8905f81e9f" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5909a1b97ed428d89e8d229b82dbe9e"
I20260925 02:43:50.142081 7417 raft_consensus.cc:3037] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:50.142197 7552 raft_consensus.cc:3037] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:50.144321 7417 raft_consensus.cc:2454] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 1.
I20260925 02:43:50.144485 7552 raft_consensus.cc:2454] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 1.
I20260925 02:43:50.144850 7201 leader_election.cc:302] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86438781b1934792b0d1f9e24153bab0, b42375594c7342c4a0e410b82148801b; no voters:
I20260925 02:43:50.145185 7631 raft_consensus.cc:2781] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:50.145632 7631 raft_consensus.cc:686] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 1 LEADER]: Becoming Leader. State: Replica: b42375594c7342c4a0e410b82148801b, State: Running, Role: LEADER
I20260925 02:43:50.146106 7631 consensus_queue.cc:237] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } }
I20260925 02:43:50.150663 7111 catalog_manager.cc:5495] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b reported cstate change: term changed from 0 to 1, leader changed from <none> to b42375594c7342c4a0e410b82148801b (127.5.56.129). New cstate: current_term: 1 leader_uuid: "b42375594c7342c4a0e410b82148801b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } health_report { overall_health: HEALTHY } } }
W20260925 02:43:50.165643 7603 tablet.cc:2369] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 02:43:50 dist-test-slave-n563 krb5kdc[7067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304228, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:50.214620 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.200903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60300 (local address 127.5.56.190:38925)
0925 02:43:50.201107 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.201112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.201609 (+ 497us) server_negotiation.cc:407] Connection header received
0925 02:43:50.201852 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.201857 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.201929 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.202032 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:50.203318 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.204063 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.205290 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.205724 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.205818 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.208948 (+ 3130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:50.208953 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:50.208956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:50.208984 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:50.211650 (+ 2666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:50.212502 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:50.212507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:50.212509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:50.212626 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:50.212998 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:50.213001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:50.213002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:50.213368 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:50.213485 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.213977 (+ 492us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.214302 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":143,"server-negotiator.queue_time_us":78}
I20260925 02:43:50.214696 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.200910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38925 (local address 127.0.0.1:60300)
0925 02:43:50.201510 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.201525 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.201672 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.202071 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.202075 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.202868 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:50.203143 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.203153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.204179 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.204182 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.205155 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.205167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.205884 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.205888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.205998 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.206640 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:50.206661 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:50.208780 (+ 2119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:50.211779 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:50.211792 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:50.211801 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:50.212387 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:50.212723 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:50.212727 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:50.212731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:50.212902 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:50.213496 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:50.213506 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:50.213709 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.213882 (+ 173us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.214178 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":343,"threads_started":1}
I20260925 02:43:50.220435 7111 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:60300:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:43:50.221565 7111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:43:50.228600 7250 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca748a87f0a45dd81030560a2ae0b74 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18f3d411a92b4228ae7e10da955a7150]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:43:50.228904 7532 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca748a87f0a45dd81030560a2ae0b74 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18f3d411a92b4228ae7e10da955a7150]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:43:50.229631 7250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca748a87f0a45dd81030560a2ae0b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:50.229214 7395 tablet_service.cc:1467] Processing CreateTablet for tablet 5ca748a87f0a45dd81030560a2ae0b74 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=18f3d411a92b4228ae7e10da955a7150]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:43:50.229848 7532 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca748a87f0a45dd81030560a2ae0b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:50.230026 7395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ca748a87f0a45dd81030560a2ae0b74. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:50.236006 7622 tablet_bootstrap.cc:492] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b: Bootstrap starting.
I20260925 02:43:50.237092 7623 tablet_bootstrap.cc:492] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0: Bootstrap starting.
I20260925 02:43:50.237121 7624 tablet_bootstrap.cc:492] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: Bootstrap starting.
I20260925 02:43:50.237627 7622 tablet_bootstrap.cc:654] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:50.238818 7623 tablet_bootstrap.cc:654] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:50.238816 7624 tablet_bootstrap.cc:654] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:50.239756 7622 tablet_bootstrap.cc:492] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b: No bootstrap required, opened a new log
I20260925 02:43:50.239912 7622 ts_tablet_manager.cc:1397] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260925 02:43:50.240818 7622 raft_consensus.cc:348] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.241076 7622 raft_consensus.cc:374] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:50.241170 7622 raft_consensus.cc:729] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42375594c7342c4a0e410b82148801b, State: Initialized, Role: FOLLOWER
I20260925 02:43:50.241276 7623 tablet_bootstrap.cc:492] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0: No bootstrap required, opened a new log
I20260925 02:43:50.241408 7623 ts_tablet_manager.cc:1397] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260925 02:43:50.241411 7622 consensus_queue.cc:260] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.241808 7631 tablet_replica.cc:439] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } }
I20260925 02:43:50.242175 7631 tablet_replica.cc:442] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:50.242038 7622 ts_tablet_manager.cc:1428] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260925 02:43:50.242157 7623 raft_consensus.cc:348] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.242411 7623 raft_consensus.cc:374] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:50.242527 7623 raft_consensus.cc:729] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86438781b1934792b0d1f9e24153bab0, State: Initialized, Role: FOLLOWER
I20260925 02:43:50.242707 7624 tablet_bootstrap.cc:492] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: No bootstrap required, opened a new log
I20260925 02:43:50.242834 7624 ts_tablet_manager.cc:1397] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260925 02:43:50.242797 7623 consensus_queue.cc:260] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.243237 7630 tablet_replica.cc:439] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } }
I20260925 02:43:50.243556 7630 tablet_replica.cc:442] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:50.243618 7624 raft_consensus.cc:348] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.243854 7624 raft_consensus.cc:374] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:50.243952 7624 raft_consensus.cc:729] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5909a1b97ed428d89e8d229b82dbe9e, State: Initialized, Role: FOLLOWER
I20260925 02:43:50.244565 7623 ts_tablet_manager.cc:1428] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260925 02:43:50.244834 7624 consensus_queue.cc:260] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.245484 7624 ts_tablet_manager.cc:1428] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:43:50.245328 7629 tablet_replica.cc:439] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } }
I20260925 02:43:50.245719 7629 tablet_replica.cc:442] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:50.276690 7629 raft_consensus.cc:482] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:43:50.276991 7629 raft_consensus.cc:504] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.278642 7629 leader_election.cc:288] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b42375594c7342c4a0e410b82148801b (127.5.56.129:36009), 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747)
I20260925 02:43:50.285550 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.279056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:49425 (local address 127.5.56.129:36009)
0925 02:43:50.279250 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.279257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.279573 (+ 316us) server_negotiation.cc:407] Connection header received
0925 02:43:50.279816 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.279823 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.279952 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.280041 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:50.280551 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.281431 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.283547 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.284521 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.284642 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.284694 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.285175 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.285281 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:43:50.285631 7491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.279048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36009 (local address 127.5.56.131:49425)
0925 02:43:50.279430 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.279463 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.279653 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.280141 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.280146 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.280173 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:50.280357 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.280368 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.281589 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.281596 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.283372 (+ 1776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.283386 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.284691 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.284699 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.284917 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.285014 (+ 97us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.285128 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.285287 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207}
I20260925 02:43:50.286311 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.279101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39747 (local address 127.5.56.131:44915)
0925 02:43:50.279853 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.279875 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.280007 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.280377 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.280381 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.280405 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:43:50.280779 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.280791 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.282277 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.282281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.283960 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.283975 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.285534 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.285542 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.285732 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.285771 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.285870 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.285998 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":432,"threads_started":1}
I20260925 02:43:50.286535 7618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.279055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:44915 (local address 127.5.56.130:39747)
0925 02:43:50.279250 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.279257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.280027 (+ 770us) server_negotiation.cc:407] Connection header received
0925 02:43:50.280119 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.280125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.280238 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.280341 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:43:50.281002 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.282131 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.284128 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.285286 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.285472 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.285549 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.286035 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.286204 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:43:50.286654 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca748a87f0a45dd81030560a2ae0b74" candidate_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b42375594c7342c4a0e410b82148801b" is_pre_election: true
I20260925 02:43:50.287079 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca748a87f0a45dd81030560a2ae0b74" candidate_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86438781b1934792b0d1f9e24153bab0" is_pre_election: true
I20260925 02:43:50.287101 7275 raft_consensus.cc:2454] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5909a1b97ed428d89e8d229b82dbe9e in term 0.
I20260925 02:43:50.287333 7417 raft_consensus.cc:2454] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5909a1b97ed428d89e8d229b82dbe9e in term 0.
I20260925 02:43:50.287767 7483 leader_election.cc:302] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 86438781b1934792b0d1f9e24153bab0, a5909a1b97ed428d89e8d229b82dbe9e, b42375594c7342c4a0e410b82148801b; no voters:
I20260925 02:43:50.288369 7629 raft_consensus.cc:2781] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:43:50.288517 7629 raft_consensus.cc:482] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:43:50.288635 7629 raft_consensus.cc:3037] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:50.291111 7629 raft_consensus.cc:504] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.291729 7629 leader_election.cc:288] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [CANDIDATE]: Term 1 election: Requested vote from peers b42375594c7342c4a0e410b82148801b (127.5.56.129:36009), 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747)
I20260925 02:43:50.292238 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca748a87f0a45dd81030560a2ae0b74" candidate_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b42375594c7342c4a0e410b82148801b"
I20260925 02:43:50.292220 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ca748a87f0a45dd81030560a2ae0b74" candidate_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86438781b1934792b0d1f9e24153bab0"
I20260925 02:43:50.292433 7275 raft_consensus.cc:3037] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:50.292451 7417 raft_consensus.cc:3037] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:50.295480 7417 raft_consensus.cc:2454] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5909a1b97ed428d89e8d229b82dbe9e in term 1.
I20260925 02:43:50.295495 7275 raft_consensus.cc:2454] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5909a1b97ed428d89e8d229b82dbe9e in term 1.
I20260925 02:43:50.295996 7483 leader_election.cc:302] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 86438781b1934792b0d1f9e24153bab0, a5909a1b97ed428d89e8d229b82dbe9e, b42375594c7342c4a0e410b82148801b; no voters:
I20260925 02:43:50.296339 7629 raft_consensus.cc:2781] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:50.296808 7629 raft_consensus.cc:686] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 LEADER]: Becoming Leader. State: Replica: a5909a1b97ed428d89e8d229b82dbe9e, State: Running, Role: LEADER
I20260925 02:43:50.297374 7629 consensus_queue.cc:237] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.300676 7646 tablet_replica.cc:439] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: TxnStatusTablet state changed. Reason: New leader a5909a1b97ed428d89e8d229b82dbe9e. Latest consensus state: current_term: 1 leader_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } }
I20260925 02:43:50.301116 7646 tablet_replica.cc:442] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:43:50.301851 7648 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:43:50.302136 7648 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:43:50.302078 7111 catalog_manager.cc:5495] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e reported cstate change: term changed from 0 to 1, leader changed from <none> to a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131). New cstate: current_term: 1 leader_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } health_report { overall_health: HEALTHY } } }
I20260925 02:43:50.334973 7641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.328695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45591 (local address 127.0.0.1:50966)
0925 02:43:50.328954 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.328971 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.329086 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.329577 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.329580 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.329590 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:43:50.329763 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.329769 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.331280 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.331283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.332264 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.332275 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.333245 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.333250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.333428 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.333482 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:43:50.334532 (+ 1050us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:43:50.334564 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.334627 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.334741 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 02:43:50.335081 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.328810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50966 (local address 127.5.56.131:45591)
0925 02:43:50.329028 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.329036 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.329064 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:43:50.329273 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.329278 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.329402 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.329546 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:43:50.329890 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.331153 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.332398 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.333074 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.333204 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.333668 (+ 464us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:43:50.334420 (+ 752us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:43:50.334567 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.334670 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.334765 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 02:43:50.343413 7417 raft_consensus.cc:1261] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Refusing update from remote peer a5909a1b97ed428d89e8d229b82dbe9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:50.343482 7275 raft_consensus.cc:1261] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Refusing update from remote peer a5909a1b97ed428d89e8d229b82dbe9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:50.344337 7646 consensus_queue.cc:1035] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:50.344669 7629 consensus_queue.cc:1035] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:50.354626 7630 tablet_replica.cc:439] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0: TxnStatusTablet state changed. Reason: New leader a5909a1b97ed428d89e8d229b82dbe9e. Latest consensus state: current_term: 1 leader_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } }
I20260925 02:43:50.354965 7630 tablet_replica.cc:442] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:50.358244 7631 tablet_replica.cc:439] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b: TxnStatusTablet state changed. Reason: New leader a5909a1b97ed428d89e8d229b82dbe9e. Latest consensus state: current_term: 1 leader_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } }
I20260925 02:43:50.358613 7631 tablet_replica.cc:442] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:50.358558 7646 tablet_replica.cc:439] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } }
I20260925 02:43:50.358825 7646 tablet_replica.cc:442] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:43:50.362306 7629 tablet_replica.cc:439] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } }
I20260925 02:43:50.362717 7629 tablet_replica.cc:442] T 5ca748a87f0a45dd81030560a2ae0b74 P a5909a1b97ed428d89e8d229b82dbe9e: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:43:50.365422 7654 mvcc.cc:205] Tried to move back new op lower bound from 7333086127477112832 to 7333086127301410816. Current Snapshot: MvccSnapshot[applied={T|T < 7333086127477112832}]
I20260925 02:43:50.369613 7651 mvcc.cc:205] Tried to move back new op lower bound from 7333086127477112832 to 7333086127301410816. Current Snapshot: MvccSnapshot[applied={T|T < 7333086127477112832 or (T in {7333086127477112832})}]
I20260925 02:43:50.369750 7631 tablet_replica.cc:439] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } }
I20260925 02:43:50.370024 7631 tablet_replica.cc:442] T 5ca748a87f0a45dd81030560a2ae0b74 P b42375594c7342c4a0e410b82148801b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:50.371743 7630 tablet_replica.cc:439] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } } }
I20260925 02:43:50.371994 7630 tablet_replica.cc:442] T 5ca748a87f0a45dd81030560a2ae0b74 P 86438781b1934792b0d1f9e24153bab0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:43:50.380252 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.372200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53186 (local address 127.5.56.129:36009)
0925 02:43:50.372380 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.372386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.372403 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:43:50.372494 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.372499 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.372631 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.372789 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:43:50.373392 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.375105 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.377059 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.377768 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.377884 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.378448 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:43:50.379134 (+ 686us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:43:50.379274 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.379788 (+ 514us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.379881 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:43:50.402649 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.396152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:59701 (local address 127.5.56.131:45591)
0925 02:43:50.396341 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.396347 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.396826 (+ 479us) server_negotiation.cc:407] Connection header received
0925 02:43:50.397018 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.397024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.397151 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.397245 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:43:50.397816 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.398897 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.400902 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.401325 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.401426 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.401964 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:43:50.402076 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:43:50.402160 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.402308 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.402373 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:43:50.402832 7662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.396014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45591 (local address 127.5.56.129:59701)
0925 02:43:50.396712 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.396731 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.396881 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.397280 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.397285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.397298 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:43:50.397646 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.397657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.399026 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.399033 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.400768 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.400782 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.401507 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.401514 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.401689 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.401763 (+ 74us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:43:50.402183 (+ 420us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:43:50.402195 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.402273 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.402448 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":336,"threads_started":1}
I20260925 02:43:50.413367 7417 raft_consensus.cc:1261] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Refusing update from remote peer b42375594c7342c4a0e410b82148801b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:50.413524 7552 raft_consensus.cc:1261] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Refusing update from remote peer b42375594c7342c4a0e410b82148801b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:50.414115 7634 consensus_queue.cc:1035] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [LEADER]: Connected to new peer: Peer: permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:50.414598 7631 consensus_queue.cc:1035] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:50.420433 7652 mvcc.cc:205] Tried to move back new op lower bound from 7333086127766745088 to 7333086126682148864. Current Snapshot: MvccSnapshot[applied={T|T < 7333086127766745088}]
I20260925 02:43:50.453622 7621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.446433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:36585 (local address 127.5.56.129:36009)
0925 02:43:50.446646 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.446653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.447386 (+ 733us) server_negotiation.cc:407] Connection header received
0925 02:43:50.447592 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.447598 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.447696 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.447784 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:43:50.448606 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.449490 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.451747 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.452398 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.452527 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.453014 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:43:50.453121 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:43:50.453190 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.453334 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.453400 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:43:50.453883 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.446506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36009 (local address 127.5.56.131:36585)
0925 02:43:50.447270 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.447296 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.447449 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.447950 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.447958 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.447978 (+ 20us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:43:50.448423 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.448437 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.449649 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.449657 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.451608 (+ 1951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.451625 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.452583 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.452590 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.452776 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.452852 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:43:50.453223 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:43:50.453234 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.453323 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.453475 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":413,"threads_started":1}
I20260925 02:43:50.501490 7111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:50.501806 7111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:50.506208 7111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab: Sending DeleteTablet for 3 replicas of tablet 441ea6ef7c294a13a9564a8905f81e9f
I20260925 02:43:50.507472 7395 tablet_service.cc:1514] Processing DeleteTablet for tablet 441ea6ef7c294a13a9564a8905f81e9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:50 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:35408
I20260925 02:43:50.508347 7532 tablet_service.cc:1514] Processing DeleteTablet for tablet 441ea6ef7c294a13a9564a8905f81e9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:50 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:50958
I20260925 02:43:50.508458 7671 tablet_replica.cc:330] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: stopping tablet replica
I20260925 02:43:50.508898 7671 raft_consensus.cc:2229] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:50.509318 7671 raft_consensus.cc:2258] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:50.509560 7250 tablet_service.cc:1514] Processing DeleteTablet for tablet 441ea6ef7c294a13a9564a8905f81e9f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:50 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:53172
I20260925 02:43:50.511511 7671 ts_tablet_manager.cc:1905] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:50.512372 7111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:43:50.513131 7111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:43:50.522027 7249 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1 (DEFAULT_TABLE table=test-table [id=c9cda2d433314f3e91ef1f0ae01bd6a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:50.522163 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1 (DEFAULT_TABLE table=test-table [id=c9cda2d433314f3e91ef1f0ae01bd6a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:50.522756 7249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:50.522804 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:50.526213 7395 tablet_service.cc:1467] Processing CreateTablet for tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1 (DEFAULT_TABLE table=test-table [id=c9cda2d433314f3e91ef1f0ae01bd6a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:50.526741 7395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:50.537328 7622 tablet_bootstrap.cc:492] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b: Bootstrap starting.
I20260925 02:43:50.538483 7624 tablet_bootstrap.cc:492] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e: Bootstrap starting.
I20260925 02:43:50.539419 7622 tablet_bootstrap.cc:654] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:50.540169 7673 tablet_replica.cc:330] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: stopping tablet replica
I20260925 02:43:50.540190 7624 tablet_bootstrap.cc:654] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:50.540866 7673 raft_consensus.cc:2229] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:43:50.541631 7672 tablet_replica.cc:330] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: stopping tablet replica
I20260925 02:43:50.542212 7672 raft_consensus.cc:2229] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:50.543134 7673 raft_consensus.cc:2258] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:50.543576 7672 raft_consensus.cc:2258] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:50.544054 7623 tablet_bootstrap.cc:492] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0: Bootstrap starting.
I20260925 02:43:50.545789 7623 tablet_bootstrap.cc:654] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:50.547719 7672 ts_tablet_manager.cc:1905] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:50.549894 7673 ts_tablet_manager.cc:1905] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:50.567509 7623 tablet_bootstrap.cc:492] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0: No bootstrap required, opened a new log
I20260925 02:43:50.567720 7623 ts_tablet_manager.cc:1397] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.008s
I20260925 02:43:50.568559 7623 raft_consensus.cc:348] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.568825 7623 raft_consensus.cc:374] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:50.568904 7623 raft_consensus.cc:729] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86438781b1934792b0d1f9e24153bab0, State: Initialized, Role: FOLLOWER
I20260925 02:43:50.569221 7623 consensus_queue.cc:260] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.572921 7623 ts_tablet_manager.cc:1428] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20260925 02:43:50.573204 7672 ts_tablet_manager.cc:1918] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:43:50.573439 7672 log.cc:1196] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/441ea6ef7c294a13a9564a8905f81e9f
I20260925 02:43:50.574091 7672 ts_tablet_manager.cc:1939] T 441ea6ef7c294a13a9564a8905f81e9f P a5909a1b97ed428d89e8d229b82dbe9e: Deleting consensus metadata
I20260925 02:43:50.574678 7622 tablet_bootstrap.cc:492] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b: No bootstrap required, opened a new log
I20260925 02:43:50.574878 7622 ts_tablet_manager.cc:1397] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b: Time spent bootstrapping tablet: real 0.038s user 0.009s sys 0.013s
I20260925 02:43:50.574699 7673 ts_tablet_manager.cc:1918] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:43:50.575675 7673 log.cc:1196] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/441ea6ef7c294a13a9564a8905f81e9f
I20260925 02:43:50.575768 7622 raft_consensus.cc:348] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.576061 7622 raft_consensus.cc:374] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:50.576154 7622 raft_consensus.cc:729] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42375594c7342c4a0e410b82148801b, State: Initialized, Role: FOLLOWER
I20260925 02:43:50.576431 7622 consensus_queue.cc:260] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.577061 7622 ts_tablet_manager.cc:1428] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:43:50.577337 7673 ts_tablet_manager.cc:1939] T 441ea6ef7c294a13a9564a8905f81e9f P b42375594c7342c4a0e410b82148801b: Deleting consensus metadata
I20260925 02:43:50.577636 7624 tablet_bootstrap.cc:492] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e: No bootstrap required, opened a new log
I20260925 02:43:50.577829 7624 ts_tablet_manager.cc:1397] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.007s
I20260925 02:43:50.578729 7624 raft_consensus.cc:348] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.578876 7098 catalog_manager.cc:4841] TS a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131:45591): tablet 441ea6ef7c294a13a9564a8905f81e9f (table test-table [id=21211a5182a44a79810987517b6c9e5a]) successfully deleted
I20260925 02:43:50.578995 7624 raft_consensus.cc:374] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:50.579103 7624 raft_consensus.cc:729] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5909a1b97ed428d89e8d229b82dbe9e, State: Initialized, Role: FOLLOWER
I20260925 02:43:50.579352 7624 consensus_queue.cc:260] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.580039 7624 ts_tablet_manager.cc:1428] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:43:50.580559 7096 catalog_manager.cc:4841] TS b42375594c7342c4a0e410b82148801b (127.5.56.129:36009): tablet 441ea6ef7c294a13a9564a8905f81e9f (table test-table [id=21211a5182a44a79810987517b6c9e5a]) successfully deleted
I20260925 02:43:50.581904 7671 ts_tablet_manager.cc:1918] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:43:50.582124 7671 log.cc:1196] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/441ea6ef7c294a13a9564a8905f81e9f
I20260925 02:43:50.582705 7671 ts_tablet_manager.cc:1939] T 441ea6ef7c294a13a9564a8905f81e9f P 86438781b1934792b0d1f9e24153bab0: Deleting consensus metadata
I20260925 02:43:50.584538 7097 catalog_manager.cc:4841] TS 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747): tablet 441ea6ef7c294a13a9564a8905f81e9f (table test-table [id=21211a5182a44a79810987517b6c9e5a]) successfully deleted
I20260925 02:43:50.610863 7663 raft_consensus.cc:482] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:43:50.611125 7663 raft_consensus.cc:504] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.612023 7663 leader_election.cc:288] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b42375594c7342c4a0e410b82148801b (127.5.56.129:36009), 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747)
I20260925 02:43:50.612607 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc54621d84c4d3ab56b2a9b7d8ff1e1" candidate_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b42375594c7342c4a0e410b82148801b" is_pre_election: true
I20260925 02:43:50.612913 7275 raft_consensus.cc:2454] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5909a1b97ed428d89e8d229b82dbe9e in term 0.
I20260925 02:43:50.612870 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc54621d84c4d3ab56b2a9b7d8ff1e1" candidate_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86438781b1934792b0d1f9e24153bab0" is_pre_election: true
I20260925 02:43:50.613121 7417 raft_consensus.cc:2454] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5909a1b97ed428d89e8d229b82dbe9e in term 0.
I20260925 02:43:50.613621 7484 leader_election.cc:302] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86438781b1934792b0d1f9e24153bab0, a5909a1b97ed428d89e8d229b82dbe9e; no voters:
I20260925 02:43:50.614104 7663 raft_consensus.cc:2781] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:43:50.614248 7663 raft_consensus.cc:482] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:43:50.614352 7663 raft_consensus.cc:3037] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:50.616695 7663 raft_consensus.cc:504] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.617866 7275 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc54621d84c4d3ab56b2a9b7d8ff1e1" candidate_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b42375594c7342c4a0e410b82148801b"
I20260925 02:43:50.617856 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1bc54621d84c4d3ab56b2a9b7d8ff1e1" candidate_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86438781b1934792b0d1f9e24153bab0"
I20260925 02:43:50.618083 7417 raft_consensus.cc:3037] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:50.618090 7275 raft_consensus.cc:3037] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:50.621387 7275 raft_consensus.cc:2454] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5909a1b97ed428d89e8d229b82dbe9e in term 1.
I20260925 02:43:50.621380 7417 raft_consensus.cc:2454] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5909a1b97ed428d89e8d229b82dbe9e in term 1.
I20260925 02:43:50.622180 7483 leader_election.cc:302] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5909a1b97ed428d89e8d229b82dbe9e, b42375594c7342c4a0e410b82148801b; no voters:
I20260925 02:43:50.622327 7663 leader_election.cc:288] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [CANDIDATE]: Term 1 election: Requested vote from peers b42375594c7342c4a0e410b82148801b (127.5.56.129:36009), 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747)
I20260925 02:43:50.622578 7646 raft_consensus.cc:2781] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:50.622747 7646 raft_consensus.cc:686] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 LEADER]: Becoming Leader. State: Replica: a5909a1b97ed428d89e8d229b82dbe9e, State: Running, Role: LEADER
I20260925 02:43:50.623050 7646 consensus_queue.cc:237] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:50.627625 7111 catalog_manager.cc:5495] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e reported cstate change: term changed from 0 to 1, leader changed from <none> to a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131). New cstate: current_term: 1 leader_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } health_report { overall_health: HEALTHY } } }
I20260925 02:43:50.674108 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.667180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50978 (local address 127.5.56.131:45591)
0925 02:43:50.667390 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.667396 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.667417 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:43:50.667607 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.667612 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.667701 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.667794 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:43:50.668252 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.669549 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.671615 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.672401 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.672564 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.673084 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:43:50.673236 (+ 152us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:43:50.673328 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.673508 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.673603 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:43:50.691039 7619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.684378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:49727 (local address 127.5.56.131:45591)
0925 02:43:50.684946 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:50.684954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:50.684969 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:43:50.685019 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:50.685023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:50.685096 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:50.685180 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:43:50.685659 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.686857 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.688815 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.689476 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.689643 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.690155 (+ 512us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:43:50.690302 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:43:50.690423 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:50.690597 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 02:43:50.690680 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":459}
I20260925 02:43:50.691371 7667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:50.684276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45591 (local address 127.5.56.131:49727)
0925 02:43:50.684571 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:50.684591 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:50.684731 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:50.685231 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:50.685245 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:50.685262 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:43:50.685511 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.685523 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.687008 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.687014 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.688606 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:50.688622 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:50.689691 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:50.689697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:50.689909 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:50.689970 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:43:50.690452 (+ 482us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:43:50.690464 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 02:43:50.690573 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:43:50.690727 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":23}
I20260925 02:43:50.703665 7275 raft_consensus.cc:1261] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Refusing update from remote peer a5909a1b97ed428d89e8d229b82dbe9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:50.703693 7417 raft_consensus.cc:1261] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Refusing update from remote peer a5909a1b97ed428d89e8d229b82dbe9e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:50.704409 7646 consensus_queue.cc:1035] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:50.705123 7646 consensus_queue.cc:1035] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:43:50.756875 7651 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:43:52.363191 7113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:52.363444 7113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:52.367491 7113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab: Sending DeleteTablet for 3 replicas of tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1
I20260925 02:43:52.368319 7395 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:52 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:35408
I20260925 02:43:52.368440 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:52 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:50958
I20260925 02:43:52.368462 7249 tablet_service.cc:1514] Processing DeleteTablet for tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:52 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:53172
I20260925 02:43:52.369076 7700 tablet_replica.cc:330] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0: stopping tablet replica
I20260925 02:43:52.369220 7701 tablet_replica.cc:330] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e: stopping tablet replica
I20260925 02:43:52.369453 7701 raft_consensus.cc:2229] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:43:52.369467 7700 raft_consensus.cc:2229] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:52.369899 7700 raft_consensus.cc:2258] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:52.370113 7701 raft_consensus.cc:2258] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:52.371304 7701 ts_tablet_manager.cc:1905] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:52.371348 7700 ts_tablet_manager.cc:1905] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:52.378575 7701 ts_tablet_manager.cc:1918] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:43:52.378739 7701 log.cc:1196] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/1bc54621d84c4d3ab56b2a9b7d8ff1e1
I20260925 02:43:52.379190 7701 ts_tablet_manager.cc:1939] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P a5909a1b97ed428d89e8d229b82dbe9e: Deleting consensus metadata
I20260925 02:43:52.380487 7098 catalog_manager.cc:4841] TS a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131:45591): tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1 (table test-table [id=c9cda2d433314f3e91ef1f0ae01bd6a9]) successfully deleted
I20260925 02:43:52.381237 7700 ts_tablet_manager.cc:1918] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:43:52.381431 7700 log.cc:1196] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/1bc54621d84c4d3ab56b2a9b7d8ff1e1
I20260925 02:43:52.381991 7700 ts_tablet_manager.cc:1939] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P 86438781b1934792b0d1f9e24153bab0: Deleting consensus metadata
I20260925 02:43:52.383831 7097 catalog_manager.cc:4841] TS 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747): tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1 (table test-table [id=c9cda2d433314f3e91ef1f0ae01bd6a9]) successfully deleted
I20260925 02:43:52.387001 7702 tablet_replica.cc:330] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b: stopping tablet replica
I20260925 02:43:52.387233 7702 raft_consensus.cc:2229] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:52.387490 7702 raft_consensus.cc:2258] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:52.388435 7702 ts_tablet_manager.cc:1905] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:52.393494 7702 ts_tablet_manager.cc:1918] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:43:52.393648 7702 log.cc:1196] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/1bc54621d84c4d3ab56b2a9b7d8ff1e1
I20260925 02:43:52.394086 7702 ts_tablet_manager.cc:1939] T 1bc54621d84c4d3ab56b2a9b7d8ff1e1 P b42375594c7342c4a0e410b82148801b: Deleting consensus metadata
I20260925 02:43:52.395197 7096 catalog_manager.cc:4841] TS b42375594c7342c4a0e410b82148801b (127.5.56.129:36009): tablet 1bc54621d84c4d3ab56b2a9b7d8ff1e1 (table test-table [id=c9cda2d433314f3e91ef1f0ae01bd6a9]) successfully deleted
Sep 25 02:43:52 dist-test-slave-n563 krb5kdc[7067](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:43:52 dist-test-slave-n563 krb5kdc[7067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:52.434002 7710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:52.419599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.5.56.190:38925)
0925 02:43:52.420199 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:52.420206 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:52.420287 (+ 81us) server_negotiation.cc:407] Connection header received
0925 02:43:52.420506 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:52.420510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:52.420580 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:52.420697 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:52.422195 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:52.423150 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:52.424462 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:52.424875 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:52.425012 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:52.428836 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:52.428842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:52.428846 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:52.428879 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:52.431422 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:52.432070 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:52.432076 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:52.432080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:52.432212 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:52.432573 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:52.432579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:52.432582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:52.432853 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:52.433127 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:52.433401 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 02:43:52.433630 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":275,"threads_started":1}
I20260925 02:43:52.437413 7113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:43:52.438032 7113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:43:52.444504 7249 tablet_service.cc:1467] Processing CreateTablet for tablet 36934208c9074e7782ab30fa3d052e68 (DEFAULT_TABLE table=test-table [id=27901f5d9fee4e0596f29c403b7b28c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:52.444521 7395 tablet_service.cc:1467] Processing CreateTablet for tablet 36934208c9074e7782ab30fa3d052e68 (DEFAULT_TABLE table=test-table [id=27901f5d9fee4e0596f29c403b7b28c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:52.444731 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 36934208c9074e7782ab30fa3d052e68 (DEFAULT_TABLE table=test-table [id=27901f5d9fee4e0596f29c403b7b28c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:52.444970 7249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36934208c9074e7782ab30fa3d052e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:52.444974 7395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36934208c9074e7782ab30fa3d052e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:52.445163 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36934208c9074e7782ab30fa3d052e68. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:52.451153 7711 tablet_bootstrap.cc:492] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b: Bootstrap starting.
I20260925 02:43:52.451952 7712 tablet_bootstrap.cc:492] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0: Bootstrap starting.
I20260925 02:43:52.453351 7711 tablet_bootstrap.cc:654] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:52.454121 7712 tablet_bootstrap.cc:654] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:52.456090 7713 tablet_bootstrap.cc:492] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e: Bootstrap starting.
I20260925 02:43:52.456703 7712 tablet_bootstrap.cc:492] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0: No bootstrap required, opened a new log
I20260925 02:43:52.456852 7712 ts_tablet_manager.cc:1397] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:43:52.457697 7712 raft_consensus.cc:348] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.457906 7713 tablet_bootstrap.cc:654] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:52.457998 7712 raft_consensus.cc:374] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:52.458113 7712 raft_consensus.cc:729] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86438781b1934792b0d1f9e24153bab0, State: Initialized, Role: FOLLOWER
I20260925 02:43:52.458374 7712 consensus_queue.cc:260] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.459353 7711 tablet_bootstrap.cc:492] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b: No bootstrap required, opened a new log
I20260925 02:43:52.459553 7711 ts_tablet_manager.cc:1397] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260925 02:43:52.460566 7711 raft_consensus.cc:348] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.460887 7711 raft_consensus.cc:374] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:52.461001 7711 raft_consensus.cc:729] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42375594c7342c4a0e410b82148801b, State: Initialized, Role: FOLLOWER
I20260925 02:43:52.461377 7711 consensus_queue.cc:260] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.463043 7712 ts_tablet_manager.cc:1428] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260925 02:43:52.463343 7711 ts_tablet_manager.cc:1428] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260925 02:43:52.466259 7713 tablet_bootstrap.cc:492] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e: No bootstrap required, opened a new log
I20260925 02:43:52.466450 7713 ts_tablet_manager.cc:1397] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260925 02:43:52.467413 7713 raft_consensus.cc:348] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.467638 7713 raft_consensus.cc:374] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:52.467733 7713 raft_consensus.cc:729] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5909a1b97ed428d89e8d229b82dbe9e, State: Initialized, Role: FOLLOWER
I20260925 02:43:52.467967 7713 consensus_queue.cc:260] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.468434 7713 ts_tablet_manager.cc:1428] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:43:52.647127 7717 raft_consensus.cc:482] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:43:52.647356 7717 raft_consensus.cc:504] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.648108 7717 leader_election.cc:288] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747), a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131:45591)
I20260925 02:43:52.648744 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36934208c9074e7782ab30fa3d052e68" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86438781b1934792b0d1f9e24153bab0" is_pre_election: true
I20260925 02:43:52.648833 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36934208c9074e7782ab30fa3d052e68" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" is_pre_election: true
I20260925 02:43:52.649019 7417 raft_consensus.cc:2454] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 0.
I20260925 02:43:52.649070 7551 raft_consensus.cc:2454] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 0.
I20260925 02:43:52.649551 7201 leader_election.cc:302] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86438781b1934792b0d1f9e24153bab0, b42375594c7342c4a0e410b82148801b; no voters:
I20260925 02:43:52.649859 7717 raft_consensus.cc:2781] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:43:52.649982 7717 raft_consensus.cc:482] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:43:52.650067 7717 raft_consensus.cc:3037] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:52.652357 7717 raft_consensus.cc:504] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.652951 7717 leader_election.cc:288] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 election: Requested vote from peers 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747), a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131:45591)
I20260925 02:43:52.653390 7417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36934208c9074e7782ab30fa3d052e68" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86438781b1934792b0d1f9e24153bab0"
I20260925 02:43:52.653614 7417 raft_consensus.cc:3037] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:52.653599 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36934208c9074e7782ab30fa3d052e68" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5909a1b97ed428d89e8d229b82dbe9e"
I20260925 02:43:52.653844 7551 raft_consensus.cc:3037] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:52.655821 7417 raft_consensus.cc:2454] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 1.
I20260925 02:43:52.656286 7551 raft_consensus.cc:2454] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 1.
I20260925 02:43:52.656303 7201 leader_election.cc:302] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 86438781b1934792b0d1f9e24153bab0, b42375594c7342c4a0e410b82148801b; no voters:
I20260925 02:43:52.656615 7717 raft_consensus.cc:2781] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:52.657130 7717 raft_consensus.cc:686] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 1 LEADER]: Becoming Leader. State: Replica: b42375594c7342c4a0e410b82148801b, State: Running, Role: LEADER
I20260925 02:43:52.657689 7717 consensus_queue.cc:237] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.660395 7113 catalog_manager.cc:5495] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b reported cstate change: term changed from 0 to 1, leader changed from <none> to b42375594c7342c4a0e410b82148801b (127.5.56.129). New cstate: current_term: 1 leader_uuid: "b42375594c7342c4a0e410b82148801b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } health_report { overall_health: UNKNOWN } } }
I20260925 02:43:52.703224 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:52.697702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53198 (local address 127.5.56.129:36009)
0925 02:43:52.698374 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:52.698424 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:52.698449 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:43:52.698513 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:52.698516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:52.698576 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:52.698676 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:43:52.699090 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:52.700068 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:52.701395 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:52.701818 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:52.701954 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:52.702422 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:43:52.702535 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:43:52.702636 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:52.702787 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 02:43:52.702867 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":391,"threads_started":1}
I20260925 02:43:52.712913 7551 raft_consensus.cc:1261] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Refusing update from remote peer b42375594c7342c4a0e410b82148801b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:52.712913 7417 raft_consensus.cc:1261] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Refusing update from remote peer b42375594c7342c4a0e410b82148801b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:52.713516 7719 consensus_queue.cc:1035] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [LEADER]: Connected to new peer: Peer: permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:52.713829 7717 consensus_queue.cc:1035] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:52.762558 7113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:52.762846 7113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:52.765650 7113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab: Sending DeleteTablet for 3 replicas of tablet 36934208c9074e7782ab30fa3d052e68
I20260925 02:43:52.766435 7249 tablet_service.cc:1514] Processing DeleteTablet for tablet 36934208c9074e7782ab30fa3d052e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:52 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:53172
I20260925 02:43:52.766539 7395 tablet_service.cc:1514] Processing DeleteTablet for tablet 36934208c9074e7782ab30fa3d052e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:52 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:35408
I20260925 02:43:52.766621 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 36934208c9074e7782ab30fa3d052e68 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:52 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:50958
I20260925 02:43:52.766780 7702 tablet_replica.cc:330] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b: stopping tablet replica
I20260925 02:43:52.766865 7701 tablet_replica.cc:330] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e: stopping tablet replica
I20260925 02:43:52.766999 7702 raft_consensus.cc:2229] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:43:52.767087 7701 raft_consensus.cc:2229] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:52.767290 7702 raft_consensus.cc:2258] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:52.767283 7701 raft_consensus.cc:2258] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:52.767572 7700 tablet_replica.cc:330] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0: stopping tablet replica
I20260925 02:43:52.767781 7700 raft_consensus.cc:2229] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:52.767997 7700 raft_consensus.cc:2258] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:52.768254 7113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60314:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:43:52.768954 7113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:43:52.769001 7702 ts_tablet_manager.cc:1905] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:52.769434 7700 ts_tablet_manager.cc:1905] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:52.770869 7701 ts_tablet_manager.cc:1905] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:52.774540 7700 ts_tablet_manager.cc:1918] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:43:52.774698 7700 log.cc:1196] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/36934208c9074e7782ab30fa3d052e68
I20260925 02:43:52.775112 7700 ts_tablet_manager.cc:1939] T 36934208c9074e7782ab30fa3d052e68 P 86438781b1934792b0d1f9e24153bab0: Deleting consensus metadata
I20260925 02:43:52.775520 7702 ts_tablet_manager.cc:1918] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:43:52.775665 7702 log.cc:1196] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/36934208c9074e7782ab30fa3d052e68
I20260925 02:43:52.776122 7702 ts_tablet_manager.cc:1939] T 36934208c9074e7782ab30fa3d052e68 P b42375594c7342c4a0e410b82148801b: Deleting consensus metadata
I20260925 02:43:52.776394 7097 catalog_manager.cc:4841] TS 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747): tablet 36934208c9074e7782ab30fa3d052e68 (table test-table [id=27901f5d9fee4e0596f29c403b7b28c4]) successfully deleted
I20260925 02:43:52.777216 7096 catalog_manager.cc:4841] TS b42375594c7342c4a0e410b82148801b (127.5.56.129:36009): tablet 36934208c9074e7782ab30fa3d052e68 (table test-table [id=27901f5d9fee4e0596f29c403b7b28c4]) successfully deleted
I20260925 02:43:52.778812 7701 ts_tablet_manager.cc:1918] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:43:52.779006 7701 log.cc:1196] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/36934208c9074e7782ab30fa3d052e68
I20260925 02:43:52.779458 7701 ts_tablet_manager.cc:1939] T 36934208c9074e7782ab30fa3d052e68 P a5909a1b97ed428d89e8d229b82dbe9e: Deleting consensus metadata
I20260925 02:43:52.780668 7098 catalog_manager.cc:4841] TS a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131:45591): tablet 36934208c9074e7782ab30fa3d052e68 (table test-table [id=27901f5d9fee4e0596f29c403b7b28c4]) successfully deleted
I20260925 02:43:52.780624 7395 tablet_service.cc:1467] Processing CreateTablet for tablet 41a60385ddb74aa19160815bde0cbd2f (DEFAULT_TABLE table=test-table [id=cbac0bf042b141b4a9e557f3792b0f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:52.781100 7395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a60385ddb74aa19160815bde0cbd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:52.781189 7531 tablet_service.cc:1467] Processing CreateTablet for tablet 41a60385ddb74aa19160815bde0cbd2f (DEFAULT_TABLE table=test-table [id=cbac0bf042b141b4a9e557f3792b0f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:52.781580 7531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a60385ddb74aa19160815bde0cbd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:52.782740 7249 tablet_service.cc:1467] Processing CreateTablet for tablet 41a60385ddb74aa19160815bde0cbd2f (DEFAULT_TABLE table=test-table [id=cbac0bf042b141b4a9e557f3792b0f4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:43:52.783106 7249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41a60385ddb74aa19160815bde0cbd2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:43:52.787492 7711 tablet_bootstrap.cc:492] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b: Bootstrap starting.
I20260925 02:43:52.787504 7712 tablet_bootstrap.cc:492] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0: Bootstrap starting.
I20260925 02:43:52.787671 7713 tablet_bootstrap.cc:492] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e: Bootstrap starting.
I20260925 02:43:52.789167 7711 tablet_bootstrap.cc:654] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:52.789196 7712 tablet_bootstrap.cc:654] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:52.789248 7713 tablet_bootstrap.cc:654] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:52.791647 7713 tablet_bootstrap.cc:492] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e: No bootstrap required, opened a new log
I20260925 02:43:52.791692 7711 tablet_bootstrap.cc:492] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b: No bootstrap required, opened a new log
I20260925 02:43:52.791807 7713 ts_tablet_manager.cc:1397] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260925 02:43:52.791882 7711 ts_tablet_manager.cc:1397] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20260925 02:43:52.792649 7711 raft_consensus.cc:348] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.792650 7713 raft_consensus.cc:348] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.792883 7711 raft_consensus.cc:374] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:52.792903 7713 raft_consensus.cc:374] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:52.792968 7711 raft_consensus.cc:729] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b42375594c7342c4a0e410b82148801b, State: Initialized, Role: FOLLOWER
I20260925 02:43:52.792997 7713 raft_consensus.cc:729] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5909a1b97ed428d89e8d229b82dbe9e, State: Initialized, Role: FOLLOWER
I20260925 02:43:52.793215 7711 consensus_queue.cc:260] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.793282 7713 consensus_queue.cc:260] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.793574 7712 tablet_bootstrap.cc:492] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0: No bootstrap required, opened a new log
I20260925 02:43:52.793733 7712 ts_tablet_manager.cc:1397] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.004s
I20260925 02:43:52.793742 7713 ts_tablet_manager.cc:1428] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:43:52.793818 7711 ts_tablet_manager.cc:1428] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:43:52.794502 7712 raft_consensus.cc:348] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.794802 7712 raft_consensus.cc:374] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:52.794902 7712 raft_consensus.cc:729] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86438781b1934792b0d1f9e24153bab0, State: Initialized, Role: FOLLOWER
I20260925 02:43:52.795184 7712 consensus_queue.cc:260] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:52.795671 7712 ts_tablet_manager.cc:1428] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260925 02:43:53.012176 7717 raft_consensus.cc:482] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:43:53.012389 7717 raft_consensus.cc:504] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:53.013075 7717 leader_election.cc:288] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747), a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131:45591)
I20260925 02:43:53.013746 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a60385ddb74aa19160815bde0cbd2f" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86438781b1934792b0d1f9e24153bab0" is_pre_election: true
I20260925 02:43:53.013746 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a60385ddb74aa19160815bde0cbd2f" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" is_pre_election: true
I20260925 02:43:53.014024 7416 raft_consensus.cc:2454] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 0.
I20260925 02:43:53.014047 7551 raft_consensus.cc:2454] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 0.
I20260925 02:43:53.014524 7201 leader_election.cc:302] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 86438781b1934792b0d1f9e24153bab0, a5909a1b97ed428d89e8d229b82dbe9e, b42375594c7342c4a0e410b82148801b; no voters:
I20260925 02:43:53.014822 7717 raft_consensus.cc:2781] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:43:53.014971 7717 raft_consensus.cc:482] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:43:53.015077 7717 raft_consensus.cc:3037] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:53.017313 7717 raft_consensus.cc:504] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:53.017992 7717 leader_election.cc:288] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 election: Requested vote from peers 86438781b1934792b0d1f9e24153bab0 (127.5.56.130:39747), a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131:45591)
I20260925 02:43:53.018602 7551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a60385ddb74aa19160815bde0cbd2f" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5909a1b97ed428d89e8d229b82dbe9e"
I20260925 02:43:53.018608 7416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41a60385ddb74aa19160815bde0cbd2f" candidate_uuid: "b42375594c7342c4a0e410b82148801b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86438781b1934792b0d1f9e24153bab0"
I20260925 02:43:53.018828 7551 raft_consensus.cc:3037] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:53.018828 7416 raft_consensus.cc:3037] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:53.021901 7416 raft_consensus.cc:2454] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 1.
I20260925 02:43:53.021901 7551 raft_consensus.cc:2454] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b42375594c7342c4a0e410b82148801b in term 1.
I20260925 02:43:53.022485 7202 leader_election.cc:302] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 86438781b1934792b0d1f9e24153bab0, a5909a1b97ed428d89e8d229b82dbe9e, b42375594c7342c4a0e410b82148801b; no voters:
I20260925 02:43:53.022902 7717 raft_consensus.cc:2781] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:53.023063 7717 raft_consensus.cc:686] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 1 LEADER]: Becoming Leader. State: Replica: b42375594c7342c4a0e410b82148801b, State: Running, Role: LEADER
I20260925 02:43:53.023365 7717 consensus_queue.cc:237] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } }
I20260925 02:43:53.026011 7112 catalog_manager.cc:5495] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b reported cstate change: term changed from 0 to 1, leader changed from <none> to b42375594c7342c4a0e410b82148801b (127.5.56.129). New cstate: current_term: 1 leader_uuid: "b42375594c7342c4a0e410b82148801b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b42375594c7342c4a0e410b82148801b" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 } health_report { overall_health: UNKNOWN } } }
I20260925 02:43:53.103076 7551 raft_consensus.cc:1261] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Refusing update from remote peer b42375594c7342c4a0e410b82148801b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:53.103550 7416 raft_consensus.cc:1261] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Refusing update from remote peer b42375594c7342c4a0e410b82148801b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:43:53.103690 7717 consensus_queue.cc:1035] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5909a1b97ed428d89e8d229b82dbe9e" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:43:53.104141 7726 consensus_queue.cc:1035] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [LEADER]: Connected to new peer: Peer: permanent_uuid: "86438781b1934792b0d1f9e24153bab0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:43:53.121959 7685 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 02:43:54.708338 7112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:54.708559 7112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60314:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:43:54.712741 7112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b18f1624cca745feb177d663df99ddab: Sending DeleteTablet for 3 replicas of tablet 41a60385ddb74aa19160815bde0cbd2f
I20260925 02:43:54.713629 7249 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a60385ddb74aa19160815bde0cbd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:54 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:53172
I20260925 02:43:54.713755 7395 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a60385ddb74aa19160815bde0cbd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:54 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:35408
I20260925 02:43:54.713790 7531 tablet_service.cc:1514] Processing DeleteTablet for tablet 41a60385ddb74aa19160815bde0cbd2f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:43:54 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:50958
I20260925 02:43:54.714496 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 7184
I20260925 02:43:54.725468 7757 tablet_replica.cc:330] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0: stopping tablet replica
I20260925 02:43:54.725934 7757 raft_consensus.cc:2229] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:54.726341 7757 raft_consensus.cc:2258] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:54.727969 7757 ts_tablet_manager.cc:1905] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:54.733978 7756 tablet_replica.cc:330] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e: stopping tablet replica
I20260925 02:43:54.734227 7756 raft_consensus.cc:2229] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:43:54.734277 7758 tablet_replica.cc:330] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b: stopping tablet replica
I20260925 02:43:54.734547 7756 raft_consensus.cc:2258] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:54.734529 7758 raft_consensus.cc:2229] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:43:54.734982 7758 raft_consensus.cc:2258] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:43:54.735987 7756 ts_tablet_manager.cc:1905] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:43:54.736116 7758 ts_tablet_manager.cc:1905] T 41a60385ddb74aa19160815bde0cbd2f P b42375594c7342c4a0e410b82148801b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 02:43:54 dist-test-slave-n563 krb5kdc[7067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:43:54.749673 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:54.716274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53202 (local address 127.5.56.129:36009)
0925 02:43:54.733714 (+ 17440us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:54.733722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:54.733751 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:43:54.733820 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:54.733826 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:54.733915 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:54.734040 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:54.736058 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:54.737409 (+ 1351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:54.739320 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:54.739740 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:54.739846 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:54.743892 (+ 4046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:54.743909 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:54.743927 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:54.743964 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:54.747053 (+ 3089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:54.747809 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:54.747817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:54.747819 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:54.747951 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:54.748358 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:54.748362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:54.748364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:54.748751 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:54.749041 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:54.749214 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 02:43:54.749399 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17265,"thread_start_us":17124,"threads_started":1}
W20260925 02:43:54.750348 7316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53202
W20260925 02:43:54.751195 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 7184: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:43:54.751423 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 7184
I20260925 02:43:54.775338 7756 ts_tablet_manager.cc:1918] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:43:54.775528 7756 log.cc:1196] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/41a60385ddb74aa19160815bde0cbd2f
I20260925 02:43:54.775995 7756 ts_tablet_manager.cc:1939] T 41a60385ddb74aa19160815bde0cbd2f P a5909a1b97ed428d89e8d229b82dbe9e: Deleting consensus metadata
I20260925 02:43:54.777254 7757 ts_tablet_manager.cc:1918] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 02:43:54.777422 7757 log.cc:1196] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/41a60385ddb74aa19160815bde0cbd2f
I20260925 02:43:54.777997 7757 ts_tablet_manager.cc:1939] T 41a60385ddb74aa19160815bde0cbd2f P 86438781b1934792b0d1f9e24153bab0: Deleting consensus metadata
I20260925 02:43:54.795382 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 7326
W20260925 02:43:54.795392 7096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.56.129:36009 (error 108)
I20260925 02:43:54.798089 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:54.797037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36009 (local address 127.0.0.1:53216)
0925 02:43:54.797598 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:54.797749 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":256,"threads_started":1}
W20260925 02:43:54.798323 7096 catalog_manager.cc:4584] TS b42375594c7342c4a0e410b82148801b (127.5.56.129:36009): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 41a60385ddb74aa19160815bde0cbd2f: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36009: connect: Connection refused (error 111)
Sep 25 02:43:54 dist-test-slave-n563 krb5kdc[7067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:43:54.811501 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:54.796455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35412 (local address 127.5.56.130:39747)
0925 02:43:54.797610 (+ 1155us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:54.797615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:54.797632 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:43:54.797686 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:54.797689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:54.797749 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:54.797839 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:54.799349 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:54.800075 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:54.801350 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:54.801759 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:54.801847 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:54.805983 (+ 4136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:54.805998 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:54.806014 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:54.806052 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:54.808850 (+ 2798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:54.809563 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:54.809570 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:54.809572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:54.809749 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:54.810140 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:54.810143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:54.810145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:54.810577 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:54.810813 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:54.811060 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 02:43:54.811235 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1029,"thread_start_us":275,"threads_started":1}
W20260925 02:43:54.812247 7457 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35412
W20260925 02:43:54.812885 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 7326: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:43:54.813020 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 7326
I20260925 02:43:54.852061 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:54.851426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36009 (local address 127.0.0.1:53228)
0925 02:43:54.851694 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:54.851821 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260925 02:43:54.853253 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:54.852743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36009 (local address 127.0.0.1:53244)
0925 02:43:54.852933 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:54.853067 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:43:54.856683 7098 catalog_manager.cc:4841] TS a5909a1b97ed428d89e8d229b82dbe9e (127.5.56.131:45591): tablet 41a60385ddb74aa19160815bde0cbd2f (table test-table [id=cbac0bf042b141b4a9e557f3792b0f4c]) successfully deleted
I20260925 02:43:54.869913 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 7466
I20260925 02:43:54.870994 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:54.870496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39747 (local address 127.0.0.1:35426)
0925 02:43:54.870696 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:54.870790 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.130:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
Sep 25 02:43:54 dist-test-slave-n563 krb5kdc[7067](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304232, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:43:54.887388 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:54.870767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50980 (local address 127.5.56.131:45591)
0925 02:43:54.871342 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:54.871347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:54.871364 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:43:54.871420 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:54.871424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:54.871496 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:54.871585 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:54.872925 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:54.873723 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:54.875017 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:54.875486 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:54.875603 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:54.880781 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:54.880797 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:54.880814 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:54.880857 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:54.884343 (+ 3486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:54.885141 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:54.885150 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:54.885153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:54.885290 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:54.885674 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:54.885680 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:54.885683 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:54.886050 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:54.886365 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:54.886770 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 02:43:54.886995 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":595,"threads_started":1}
W20260925 02:43:54.888069 7597 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:50980
W20260925 02:43:54.888706 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 7466: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:43:54.888901 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 7466
I20260925 02:43:54.903851 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:54.903357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39747 (local address 127.0.0.1:35438)
0925 02:43:54.903583 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:54.903682 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.130:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 02:43:54.904939 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:54.904576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39747 (local address 127.0.0.1:35450)
0925 02:43:54.904756 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:54.904827 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.130:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:43:54.913852 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:54.913373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36009 (local address 127.0.0.1:53250)
0925 02:43:54.913591 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:54.913694 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260925 02:43:54.914951 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:54.914582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36009 (local address 127.0.0.1:53258)
0925 02:43:54.914771 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:54.914842 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36009: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 02:43:54.917680 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 7080
I20260925 02:43:54.918785 7153 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:43:55.041685 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:55.040561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39747 (local address 127.0.0.1:35452)
0925 02:43:55.041307 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:55.041447 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.130:39747: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":564}
I20260925 02:43:55.041756 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 7080
2026-09-25T02:43:55Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8549 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:43:56 dist-test-slave-n563 krb5kdc[7770](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:43:56 dist-test-slave-n563 krb5kdc[7770](info): set up 2 sockets
Sep 25 02:43:56 dist-test-slave-n563 krb5kdc[7770](info): commencing operation
krb5kdc: starting...
W20260925 02:43:57.091871 5346 mini_kdc.cc:121] Time spent starting KDC: real 2.003s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:43:57 dist-test-slave-n563 krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:43:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:43:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:43:57.301874 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:38303
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:38303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:57.633873 7783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:57.634294 7783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:57.634483 7783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:57.642218 7783 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:43:57.642392 7783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:57.642515 7783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:57.642611 7783 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:43:57.642689 7783 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:43:57.652972 7783 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:38303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:38303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:57.654467 7783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:57.655851 7783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:57.668458 7788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:57.668533 7791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:57.668650 7789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:57.669068 7783 server_base.cc:1034] running on GCE node
I20260925 02:43:57.670097 7783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:57.671640 7783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:57.672869 7783 hybrid_clock.cc:648] HybridClock initialized: now 1790304237672806 us; error 66 us; skew 500 ppm
I20260925 02:43:57.673231 7783 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:57 dist-test-slave-n563 krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304237, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:57.676452 7783 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:43:57.677969 7783 webserver.cc:460] Webserver started at http://127.5.56.190:38351/ using document root <none> and password file <none>
I20260925 02:43:57.678745 7783 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:57.678882 7783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:57.679203 7783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:57.682228 7783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "9edbb958c35743a5a4e32bd3e724dd83"
format_stamp: "Formatted at 2026-09-25 02:43:57 on dist-test-slave-n563"
server_key: "6d2093f6fe2237b9db828cf2d73944c0"
server_key_iv: "7bbb433563d3c731114b4967dc39cd92"
server_key_version: "encryptionkey@0"
I20260925 02:43:57.683116 7783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "9edbb958c35743a5a4e32bd3e724dd83"
format_stamp: "Formatted at 2026-09-25 02:43:57 on dist-test-slave-n563"
server_key: "6d2093f6fe2237b9db828cf2d73944c0"
server_key_iv: "7bbb433563d3c731114b4967dc39cd92"
server_key_version: "encryptionkey@0"
I20260925 02:43:57.688397 7783 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:43:57.692587 7798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:57.694101 7783 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:43:57.694315 7783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "9edbb958c35743a5a4e32bd3e724dd83"
format_stamp: "Formatted at 2026-09-25 02:43:57 on dist-test-slave-n563"
server_key: "6d2093f6fe2237b9db828cf2d73944c0"
server_key_iv: "7bbb433563d3c731114b4967dc39cd92"
server_key_version: "encryptionkey@0"
I20260925 02:43:57.694608 7783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:57.724197 7783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:57.728739 7783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:57.729044 7783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:57.748356 7783 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:38303
I20260925 02:43:57.748376 7849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:38303 every 8 connection(s)
I20260925 02:43:57.749811 7783 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:43:57.751533 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 7783
I20260925 02:43:57.751770 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:43:57.752607 5346 external_mini_cluster.cc:1428] Setting key 470ab9dcd4081d93f1a8a6d8fd136eea
I20260925 02:43:57.754788 7850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:43:57 dist-test-slave-n563 krb5kdc[7770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:57.768847 7850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83: Bootstrap starting.
I20260925 02:43:57.772228 7850 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83: Neither blocks nor log segments found. Creating new log.
I20260925 02:43:57.773316 7850 log.cc:824] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83: Log is configured to *not* fsync() on all Append() calls
I20260925 02:43:57.776201 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:57.754827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47820 (local address 127.5.56.190:38303)
0925 02:43:57.755677 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:57.755690 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:57.755731 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:43:57.757340 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:57.757361 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:57.757764 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:57.758095 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:57.760068 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:57.761420 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:57.763238 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:57.763924 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:57.764067 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:57.768765 (+ 4698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:57.768773 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:57.768790 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:57.768833 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:57.771962 (+ 3129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:57.772767 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:57.772774 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:57.772780 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:57.772946 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:57.773351 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:57.773355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:57.773357 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:57.773945 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:57.774276 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:57.774776 (+ 500us) server_negotiation.cc:299] Negotiation successful
0925 02:43:57.775206 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":615,"thread_start_us":369,"threads_started":1}
I20260925 02:43:57.776544 7850 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83: No bootstrap required, opened a new log
I20260925 02:43:57.784258 7850 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9edbb958c35743a5a4e32bd3e724dd83" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38303 } }
I20260925 02:43:57.784714 7850 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:43:57.784837 7850 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9edbb958c35743a5a4e32bd3e724dd83, State: Initialized, Role: FOLLOWER
I20260925 02:43:57.785429 7850 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9edbb958c35743a5a4e32bd3e724dd83" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38303 } }
I20260925 02:43:57.785676 7850 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:43:57.785851 7850 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:43:57.786028 7850 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:43:57.788271 7850 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9edbb958c35743a5a4e32bd3e724dd83" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38303 } }
I20260925 02:43:57.788828 7850 leader_election.cc:302] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9edbb958c35743a5a4e32bd3e724dd83; no voters:
I20260925 02:43:57.789440 7850 leader_election.cc:288] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:43:57.789601 7855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:43:57.790333 7855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [term 1 LEADER]: Becoming Leader. State: Replica: 9edbb958c35743a5a4e32bd3e724dd83, State: Running, Role: LEADER
I20260925 02:43:57.790922 7855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9edbb958c35743a5a4e32bd3e724dd83" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38303 } }
I20260925 02:43:57.791703 7850 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:43:57.794462 7856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9edbb958c35743a5a4e32bd3e724dd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9edbb958c35743a5a4e32bd3e724dd83" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38303 } } }
I20260925 02:43:57.794934 7856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [sys.catalog]: This master's current role is: LEADER
I20260925 02:43:57.794840 7857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9edbb958c35743a5a4e32bd3e724dd83. Latest consensus state: current_term: 1 leader_uuid: "9edbb958c35743a5a4e32bd3e724dd83" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9edbb958c35743a5a4e32bd3e724dd83" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38303 } } }
I20260925 02:43:57.795135 7857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83 [sys.catalog]: This master's current role is: LEADER
I20260925 02:43:57.796768 7864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:43:57.801748 7864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:43:57.810237 7864 catalog_manager.cc:1347] Generated new cluster ID: d18caf31039d44a7b66887dc258bfaab
I20260925 02:43:57.810371 7864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:43:57.822587 7864 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:43:57.823942 7864 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:43:57.834574 7864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9edbb958c35743a5a4e32bd3e724dd83: Generated new TSK 0
I20260925 02:43:57.835527 7864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:43:57.883947 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38303
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:58.223029 7876 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:58.223332 7876 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:58.223488 7876 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:58.230983 7876 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:58.231117 7876 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:58.231302 7876 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:43:58.241659 7876 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:58.243103 7876 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:58.244390 7876 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:58.257989 7881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:58.258047 7882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:58.257958 7884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:58.259454 7876 server_base.cc:1034] running on GCE node
I20260925 02:43:58.260166 7876 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:58.261909 7876 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:58.263100 7876 hybrid_clock.cc:648] HybridClock initialized: now 1790304238263040 us; error 46 us; skew 500 ppm
I20260925 02:43:58.263497 7876 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:58 dist-test-slave-n563 krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:58.266901 7876 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:43:58.268500 7876 webserver.cc:460] Webserver started at http://127.5.56.129:34887/ using document root <none> and password file <none>
I20260925 02:43:58.269230 7876 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:58.269356 7876 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:58.269697 7876 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:58.272804 7876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "5b695d9c06624667867897c5808eb7f7"
format_stamp: "Formatted at 2026-09-25 02:43:58 on dist-test-slave-n563"
server_key: "7261f93ff90a40f0ecb0e5e60a8d041a"
server_key_iv: "67a39cf3f71a0255b60f8d203e7f9ba2"
server_key_version: "encryptionkey@0"
I20260925 02:43:58.273600 7876 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "5b695d9c06624667867897c5808eb7f7"
format_stamp: "Formatted at 2026-09-25 02:43:58 on dist-test-slave-n563"
server_key: "7261f93ff90a40f0ecb0e5e60a8d041a"
server_key_iv: "67a39cf3f71a0255b60f8d203e7f9ba2"
server_key_version: "encryptionkey@0"
I20260925 02:43:58.279511 7876 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260925 02:43:58.283171 7891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:58.284564 7876 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:43:58.284749 7876 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "5b695d9c06624667867897c5808eb7f7"
format_stamp: "Formatted at 2026-09-25 02:43:58 on dist-test-slave-n563"
server_key: "7261f93ff90a40f0ecb0e5e60a8d041a"
server_key_iv: "67a39cf3f71a0255b60f8d203e7f9ba2"
server_key_version: "encryptionkey@0"
I20260925 02:43:58.284971 7876 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:58.303750 7876 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:58.308002 7876 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:58.308317 7876 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:58.309296 7876 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:43:58.310753 7876 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:58.310853 7876 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:58.310994 7876 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:58.311069 7876 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:58.345521 7876 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:40955
I20260925 02:43:58.345613 8004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:40955 every 8 connection(s)
I20260925 02:43:58.346935 7876 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:43:58.354291 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 7876
I20260925 02:43:58.354566 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:43:58.355357 5346 external_mini_cluster.cc:1428] Setting key 584bd315d3206adac69acfcc20a72e30
Sep 25 02:43:58 dist-test-slave-n563 krb5kdc[7770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:58.369383 8007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:58.349546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:42223 (local address 127.5.56.190:38303)
0925 02:43:58.350059 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:58.350063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:58.351203 (+ 1140us) server_negotiation.cc:407] Connection header received
0925 02:43:58.353074 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:58.353080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:58.353173 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:58.353288 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:58.355359 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:58.356200 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:58.358505 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:58.358898 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:58.358998 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:58.363464 (+ 4466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:58.363470 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:58.363473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:58.363508 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:58.366179 (+ 2671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:58.366997 (+ 818us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:58.367008 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:58.367011 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:58.367131 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:58.367577 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:58.367581 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:58.367593 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:58.367867 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:58.368023 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:58.368774 (+ 751us) server_negotiation.cc:299] Negotiation successful
0925 02:43:58.369013 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":230,"threads_started":1}
I20260925 02:43:58.370577 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:58.350149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38303 (local address 127.5.56.129:42223)
0925 02:43:58.350935 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:58.350976 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:58.352796 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:58.353532 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:58.353540 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:58.354274 (+ 734us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:58.355165 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:58.355195 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:58.356439 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:58.356445 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:58.358323 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:58.358340 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:58.359178 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:58.359189 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:58.359552 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:58.360372 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:58.360412 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:58.363207 (+ 2795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:58.366341 (+ 3134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:58.366352 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:58.366373 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:58.366859 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:58.367241 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:58.367246 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:58.367250 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:58.367457 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:58.367998 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:58.368008 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:58.368389 (+ 381us) client_negotiation.cc:769] Sending connection context
0925 02:43:58.368662 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 02:43:58.369247 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":337,"threads_started":1}
I20260925 02:43:58.372145 8005 heartbeater.cc:347] Connected to a master server at 127.5.56.190:38303
I20260925 02:43:58.372534 8005 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:58.373569 8005 heartbeater.cc:511] Master 127.5.56.190:38303 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
I20260925 02:43:58.376108 7815 ts_manager.cc:194] Registered new tserver with Master: 5b695d9c06624667867897c5808eb7f7 (127.5.56.129:40955)
I20260925 02:43:58.377645 7815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:42223
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:43:58.402007 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38303
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:58.740060 8011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:58.740420 8011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:58.740602 8011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:58.748338 8011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:58.748507 8011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:58.748685 8011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:43:58.758985 8011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:58.760350 8011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:58.761749 8011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:58.775756 8016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:58.775792 8019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:58.775700 8017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:58.777076 8011 server_base.cc:1034] running on GCE node
I20260925 02:43:58.777756 8011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:58.778860 8011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:58.780129 8011 hybrid_clock.cc:648] HybridClock initialized: now 1790304238780072 us; error 47 us; skew 500 ppm
I20260925 02:43:58.780508 8011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:58 dist-test-slave-n563 krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:58.783991 8011 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:43:58.785635 8011 webserver.cc:460] Webserver started at http://127.5.56.130:40865/ using document root <none> and password file <none>
I20260925 02:43:58.786372 8011 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:58.786602 8011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:58.786963 8011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:58.790198 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "0d25177e1d9240c6a9e8bd16653c581b"
format_stamp: "Formatted at 2026-09-25 02:43:58 on dist-test-slave-n563"
server_key: "7622b59934928ae1cfc48b15142a55ef"
server_key_iv: "564c55bf8ef82dae7eb025468efdeb14"
server_key_version: "encryptionkey@0"
I20260925 02:43:58.791097 8011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "0d25177e1d9240c6a9e8bd16653c581b"
format_stamp: "Formatted at 2026-09-25 02:43:58 on dist-test-slave-n563"
server_key: "7622b59934928ae1cfc48b15142a55ef"
server_key_iv: "564c55bf8ef82dae7eb025468efdeb14"
server_key_version: "encryptionkey@0"
I20260925 02:43:58.797149 8011 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:43:58.801177 8026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:58.802701 8011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:43:58.802901 8011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "0d25177e1d9240c6a9e8bd16653c581b"
format_stamp: "Formatted at 2026-09-25 02:43:58 on dist-test-slave-n563"
server_key: "7622b59934928ae1cfc48b15142a55ef"
server_key_iv: "564c55bf8ef82dae7eb025468efdeb14"
server_key_version: "encryptionkey@0"
I20260925 02:43:58.803136 8011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:58.830039 8011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:58.834434 8011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:58.834827 8011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:58.835927 8011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:43:58.837339 8011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:58.837450 8011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:58.837568 8011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:58.837659 8011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:58.872498 8011 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:34483
I20260925 02:43:58.872553 8140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:34483 every 8 connection(s)
I20260925 02:43:58.873821 8011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:43:58.882687 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 8011
I20260925 02:43:58.882923 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:43:58.883387 5346 external_mini_cluster.cc:1428] Setting key 5c089fb31eb8a0cbe5eea13f3e007fc5
Sep 25 02:43:58 dist-test-slave-n563 krb5kdc[7770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304238, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:58.896852 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:58.876576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:48767 (local address 127.5.56.190:38303)
0925 02:43:58.877049 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:58.877054 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:58.877900 (+ 846us) server_negotiation.cc:407] Connection header received
0925 02:43:58.879883 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:58.879889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:58.879995 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:58.880119 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:58.881726 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:58.882713 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:58.884489 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:58.884982 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:58.885135 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:58.889492 (+ 4357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:58.889501 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:58.889508 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:58.889554 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:58.893156 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:58.893996 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:58.894005 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:58.894011 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:58.894187 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:58.894684 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:58.894692 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:58.894697 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:58.895111 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:58.895268 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:58.896017 (+ 749us) server_negotiation.cc:299] Negotiation successful
0925 02:43:58.896295 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":263,"threads_started":1}
I20260925 02:43:58.897802 8144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:58.877010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38303 (local address 127.5.56.130:48767)
0925 02:43:58.877728 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:58.877773 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:58.879599 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:58.880378 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:58.880388 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:58.880950 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:58.881548 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:58.881566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:58.882936 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:58.882940 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:58.884318 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:58.884336 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:58.885233 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:58.885243 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:58.885484 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:58.886200 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:58.886223 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:58.889133 (+ 2910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:58.893338 (+ 4205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:58.893345 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:58.893361 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:58.893858 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:58.894302 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:58.894305 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:58.894307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:58.894574 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:58.895240 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:58.895247 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:58.895624 (+ 377us) client_negotiation.cc:769] Sending connection context
0925 02:43:58.895873 (+ 249us) client_negotiation.cc:240] Negotiation successful
0925 02:43:58.896385 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":282,"threads_started":1}
I20260925 02:43:58.899505 8141 heartbeater.cc:347] Connected to a master server at 127.5.56.190:38303
I20260925 02:43:58.899878 8141 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:58.900640 8141 heartbeater.cc:511] Master 127.5.56.190:38303 requested a full tablet report, sending...
I20260925 02:43:58.902143 7815 ts_manager.cc:194] Registered new tserver with Master: 0d25177e1d9240c6a9e8bd16653c581b (127.5.56.130:34483)
I20260925 02:43:58.903051 7815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:48767
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:43:58.930263 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38303
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:43:59.273064 8147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:43:59.273417 8147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:43:59.273597 8147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:43:59.281055 8147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:43:59.281230 8147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:43:59.281409 8147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:43:59.291785 8147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38303
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:43:59.293339 8147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:43:59.294804 8147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:43:59.309314 8152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:59.309270 8155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:43:59.309360 8153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:43:59.311933 8147 server_base.cc:1034] running on GCE node
I20260925 02:43:59.312626 8147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:43:59.313649 8147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:43:59.314875 8147 hybrid_clock.cc:648] HybridClock initialized: now 1790304239314829 us; error 44 us; skew 500 ppm
I20260925 02:43:59.315217 8147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:43:59 dist-test-slave-n563 krb5kdc[7770](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304239, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:43:59.324213 8147 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:43:59.325999 8147 webserver.cc:460] Webserver started at http://127.5.56.131:41033/ using document root <none> and password file <none>
I20260925 02:43:59.326869 8147 fs_manager.cc:359] Metadata directory not provided
I20260925 02:43:59.327023 8147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:43:59.327365 8147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:43:59.330704 8147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "457231c7636f4602976fc5fc4c52f0e1"
format_stamp: "Formatted at 2026-09-25 02:43:59 on dist-test-slave-n563"
server_key: "3638c29292eff7fd25a780cde5ce0e00"
server_key_iv: "4a8d209543eb9b36700d1d912fab115c"
server_key_version: "encryptionkey@0"
I20260925 02:43:59.331538 8147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "457231c7636f4602976fc5fc4c52f0e1"
format_stamp: "Formatted at 2026-09-25 02:43:59 on dist-test-slave-n563"
server_key: "3638c29292eff7fd25a780cde5ce0e00"
server_key_iv: "4a8d209543eb9b36700d1d912fab115c"
server_key_version: "encryptionkey@0"
I20260925 02:43:59.336942 8147 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:43:59.340691 8162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:59.342120 8147 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:43:59.342305 8147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "457231c7636f4602976fc5fc4c52f0e1"
format_stamp: "Formatted at 2026-09-25 02:43:59 on dist-test-slave-n563"
server_key: "3638c29292eff7fd25a780cde5ce0e00"
server_key_iv: "4a8d209543eb9b36700d1d912fab115c"
server_key_version: "encryptionkey@0"
I20260925 02:43:59.342576 8147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:43:59.360293 8147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:43:59.364455 8147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:43:59.364763 8147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:43:59.365813 8147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:43:59.367326 8147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:43:59.367432 8147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:59.367590 8147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:43:59.367664 8147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:43:59.380776 8005 heartbeater.cc:503] Master 127.5.56.190:38303 was elected leader, sending a full tablet report...
I20260925 02:43:59.403913 8147 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:35545
I20260925 02:43:59.404004 8276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:35545 every 8 connection(s)
I20260925 02:43:59.405268 8147 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:43:59.411168 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 8147
I20260925 02:43:59.411365 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:43:59.411799 5346 external_mini_cluster.cc:1428] Setting key 1c12e8b8b8c5ddd70f8daae7cfe4242a
Sep 25 02:43:59 dist-test-slave-n563 krb5kdc[7770](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304239, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:43:59.426965 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:59.408162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:57219 (local address 127.5.56.190:38303)
0925 02:43:59.408628 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:59.408633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:59.409563 (+ 930us) server_negotiation.cc:407] Connection header received
0925 02:43:59.411611 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:59.411617 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:59.411727 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:59.411849 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:59.413846 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:59.414728 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:59.416429 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:43:59.416846 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:43:59.416986 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:59.420610 (+ 3624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:43:59.420615 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:43:59.420618 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:43:59.420651 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:43:59.423289 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:59.424042 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:59.424046 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:59.424048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:59.424199 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:43:59.424596 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:43:59.424599 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:43:59.424601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:43:59.424857 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:43:59.424975 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:43:59.426245 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0925 02:43:59.426538 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":241,"threads_started":1}
I20260925 02:43:59.428064 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:43:59.408659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38303 (local address 127.5.56.131:57219)
0925 02:43:59.409397 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0925 02:43:59.409436 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:43:59.411287 (+ 1851us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:43:59.412108 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:43:59.412117 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:43:59.412805 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:43:59.413519 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:59.413540 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:59.414895 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:59.414899 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:59.416275 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:43:59.416288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:43:59.417081 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:43:59.417089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:43:59.417338 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:43:59.418014 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:43:59.418043 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:43:59.420394 (+ 2351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:43:59.423439 (+ 3045us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:59.423446 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:59.423461 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:59.423935 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:59.424297 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:43:59.424300 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:43:59.424302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:43:59.424501 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:43:59.425015 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:43:59.425023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:43:59.425620 (+ 597us) client_negotiation.cc:769] Sending connection context
0925 02:43:59.426117 (+ 497us) client_negotiation.cc:240] Negotiation successful
0925 02:43:59.426733 (+ 616us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":303,"threads_started":1}
I20260925 02:43:59.429426 8277 heartbeater.cc:347] Connected to a master server at 127.5.56.190:38303
I20260925 02:43:59.429742 8277 heartbeater.cc:464] Registering TS with master...
I20260925 02:43:59.430550 8277 heartbeater.cc:511] Master 127.5.56.190:38303 requested a full tablet report, sending...
I20260925 02:43:59.432008 7815 ts_manager.cc:194] Registered new tserver with Master: 457231c7636f4602976fc5fc4c52f0e1 (127.5.56.131:35545)
I20260925 02:43:59.432833 7815 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:57219
I20260925 02:43:59.440176 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:43:59.472822 8279 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:43:59.470303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47836 (local address 127.5.56.190:38303)
0925 02:43:59.470545 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:43:59.470550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:43:59.471123 (+ 573us) server_negotiation.cc:407] Connection header received
0925 02:43:59.471256 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:43:59.471261 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:43:59.471352 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:43:59.471459 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:43:59.472632 (+ 1173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47836: BlockingRecv error: recv got EOF from 127.0.0.1:47836 (error 108)
Metrics: {"server-negotiator.queue_time_us":122}
I20260925 02:43:59.475968 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 7876
W20260925 02:43:59.479089 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 7876: Not authorized: Client connection negotiation failed: client connection to 127.5.56.129:40955: server requires authentication, but client does not have Kerberos credentials available
I20260925 02:43:59.479254 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 7876
I20260925 02:43:59.496917 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8011
W20260925 02:43:59.499919 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8011: Not authorized: Client connection negotiation failed: client connection to 127.5.56.130:34483: server requires authentication, but client does not have Kerberos credentials available
I20260925 02:43:59.500101 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 8011
I20260925 02:43:59.514302 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8147
W20260925 02:43:59.517493 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8147: Not authorized: Client connection negotiation failed: client connection to 127.5.56.131:35545: server requires authentication, but client does not have Kerberos credentials available
I20260925 02:43:59.517618 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 8147
I20260925 02:43:59.534205 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 7783
I20260925 02:43:59.535688 7845 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:43:59.640021 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 7783
2026-09-25T02:43:59Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4593 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:43:59 dist-test-slave-n563 krb5kdc[8295](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:43:59 dist-test-slave-n563 krb5kdc[8295](info): set up 2 sockets
Sep 25 02:43:59 dist-test-slave-n563 krb5kdc[8295](info): commencing operation
krb5kdc: starting...
W20260925 02:44:00.736030 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:44:00 dist-test-slave-n563 krb5kdc[8295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:44:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:44:00.943859 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:46421
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:37343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:46421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:01.290593 8308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:01.290943 8308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:01.291052 8308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:01.298767 8308 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:01.298902 8308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:01.298949 8308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:01.299006 8308 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:01.299055 8308 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:01.309559 8308 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:37343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:46421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:46421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:01.311005 8308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:01.312331 8308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:01.325129 8313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:01.325084 8316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:01.325029 8314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:01.325984 8308 server_base.cc:1034] running on GCE node
I20260925 02:44:01.326927 8308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:01.328469 8308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:01.329834 8308 hybrid_clock.cc:648] HybridClock initialized: now 1790304241329766 us; error 66 us; skew 500 ppm
I20260925 02:44:01.330257 8308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:01 dist-test-slave-n563 krb5kdc[8295](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304241, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:01.333742 8308 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:44:01.335433 8308 webserver.cc:460] Webserver started at http://127.5.56.190:42835/ using document root <none> and password file <none>
I20260925 02:44:01.336166 8308 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:01.336313 8308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:01.336649 8308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:01.339871 8308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "93f0cf8ab2a540d59989939c3c6fc4fc"
format_stamp: "Formatted at 2026-09-25 02:44:01 on dist-test-slave-n563"
server_key: "5ba4898c4cd287f8c7f931edb7f4b77a"
server_key_iv: "7c630f513e8884bec275ee5b4ce56c05"
server_key_version: "encryptionkey@0"
I20260925 02:44:01.340696 8308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "93f0cf8ab2a540d59989939c3c6fc4fc"
format_stamp: "Formatted at 2026-09-25 02:44:01 on dist-test-slave-n563"
server_key: "5ba4898c4cd287f8c7f931edb7f4b77a"
server_key_iv: "7c630f513e8884bec275ee5b4ce56c05"
server_key_version: "encryptionkey@0"
I20260925 02:44:01.346432 8308 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:44:01.350220 8323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:01.351770 8308 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 02:44:01.351985 8308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "93f0cf8ab2a540d59989939c3c6fc4fc"
format_stamp: "Formatted at 2026-09-25 02:44:01 on dist-test-slave-n563"
server_key: "5ba4898c4cd287f8c7f931edb7f4b77a"
server_key_iv: "7c630f513e8884bec275ee5b4ce56c05"
server_key_version: "encryptionkey@0"
I20260925 02:44:01.352213 8308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:01.371744 8308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:01.376368 8308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:01.376703 8308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:01.396296 8308 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:46421
I20260925 02:44:01.396334 8374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:46421 every 8 connection(s)
I20260925 02:44:01.397805 8308 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:01.401737 8375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:01.405368 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 8308
I20260925 02:44:01.405606 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:01.406446 5346 external_mini_cluster.cc:1428] Setting key 718ea3a666f8add2edd31bc79dde9d50
I20260925 02:44:01.415583 8375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc: Bootstrap starting.
I20260925 02:44:01.419816 8375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:01.421289 8375 log.cc:824] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:44:01 dist-test-slave-n563 krb5kdc[8295](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:01.424674 8375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc: No bootstrap required, opened a new log
I20260925 02:44:01.431471 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:01.408554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45536 (local address 127.5.56.190:46421)
0925 02:44:01.409379 (+ 825us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:01.409393 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:01.409435 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:44:01.411228 (+ 1793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:01.411253 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:01.411705 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:01.412079 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:01.414036 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:01.415526 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:01.417322 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:01.418084 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:01.418225 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:01.423424 (+ 5199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:01.423435 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:01.423453 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:01.423495 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:01.426965 (+ 3470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:01.427749 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:01.427756 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:01.427763 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:01.427950 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:01.428347 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:01.428352 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:01.428361 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:01.428977 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:01.429333 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:01.429795 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 02:44:01.430273 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":389,"threads_started":1}
I20260925 02:44:01.433151 8375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0cf8ab2a540d59989939c3c6fc4fc" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46421 } }
I20260925 02:44:01.433627 8375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:01.433748 8375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93f0cf8ab2a540d59989939c3c6fc4fc, State: Initialized, Role: FOLLOWER
I20260925 02:44:01.434492 8375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0cf8ab2a540d59989939c3c6fc4fc" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46421 } }
I20260925 02:44:01.434747 8375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:01.434890 8375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:01.435132 8375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:01.435861 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:44:01.437559 8375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0cf8ab2a540d59989939c3c6fc4fc" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46421 } }
I20260925 02:44:01.438118 8375 leader_election.cc:302] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93f0cf8ab2a540d59989939c3c6fc4fc; no voters:
I20260925 02:44:01.438778 8375 leader_election.cc:288] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:01.438944 8380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:01.440940 8375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:01.444803 8380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [term 1 LEADER]: Becoming Leader. State: Replica: 93f0cf8ab2a540d59989939c3c6fc4fc, State: Running, Role: LEADER
I20260925 02:44:01.445437 8380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0cf8ab2a540d59989939c3c6fc4fc" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46421 } }
I20260925 02:44:01.448534 8381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93f0cf8ab2a540d59989939c3c6fc4fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0cf8ab2a540d59989939c3c6fc4fc" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46421 } } }
I20260925 02:44:01.449412 8382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93f0cf8ab2a540d59989939c3c6fc4fc. Latest consensus state: current_term: 1 leader_uuid: "93f0cf8ab2a540d59989939c3c6fc4fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93f0cf8ab2a540d59989939c3c6fc4fc" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46421 } } }
I20260925 02:44:01.449837 8382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:01.450269 8381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:01.452400 8390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:01.459299 8390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:01.480499 8390 catalog_manager.cc:1347] Generated new cluster ID: 1b74a7f96f4240ffa2eee4bcdef2f863
I20260925 02:44:01.480629 8390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:01.490017 8378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:01.482123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.5.56.190:46421)
0925 02:44:01.482280 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:01.482285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:01.483096 (+ 811us) server_negotiation.cc:407] Connection header received
0925 02:44:01.483325 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:01.483331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:01.483408 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:01.483519 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:01.484573 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:01.485351 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:01.486713 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:01.487167 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:01.487313 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:01.488477 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:01.488483 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:01.488486 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:01.489179 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:01.489249 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:01.489271 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:01.489401 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:01.489545 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 02:44:01.489667 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:44:01.500376 8390 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:01.502312 8390 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:01.515689 8390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93f0cf8ab2a540d59989939c3c6fc4fc: Generated new TSK 0
I20260925 02:44:01.516707 8390 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:44:01.529402 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8308
I20260925 02:44:01.530972 8370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:01.624256 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 8308
2026-09-25T02:44:01Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1978 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:44:01 dist-test-slave-n563 krb5kdc[8410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:44:01 dist-test-slave-n563 krb5kdc[8410](info): set up 2 sockets
Sep 25 02:44:01 dist-test-slave-n563 krb5kdc[8410](info): commencing operation
krb5kdc: starting...
W20260925 02:44:02.719765 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:44:02 dist-test-slave-n563 krb5kdc[8410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:44:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:44:02.921698 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43321
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:43321
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:03.253408 8423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:03.253788 8423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:03.253894 8423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:03.261210 8423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:03.261332 8423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:03.261379 8423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:03.261446 8423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:03.261490 8423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:03.271582 8423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:43321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:03.272925 8423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:03.274209 8423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:03.287310 8431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:03.287343 8429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:03.287897 8428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:03.288156 8423 server_base.cc:1034] running on GCE node
I20260925 02:44:03.289070 8423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:03.290669 8423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:03.291882 8423 hybrid_clock.cc:648] HybridClock initialized: now 1790304243291815 us; error 54 us; skew 500 ppm
I20260925 02:44:03.292245 8423 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:03 dist-test-slave-n563 krb5kdc[8410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304243, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:03.295637 8423 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:44:03.297183 8423 webserver.cc:460] Webserver started at http://127.5.56.190:35031/ using document root <none> and password file <none>
I20260925 02:44:03.297930 8423 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:03.298071 8423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:03.298470 8423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:03.302089 8423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "3d6099aab36d4169978b033571a7b1b9"
format_stamp: "Formatted at 2026-09-25 02:44:03 on dist-test-slave-n563"
server_key: "2bd6d62120d7f1adeafc021bc67b9438"
server_key_iv: "18de6f7db51813d170eb0628df3b17b5"
server_key_version: "encryptionkey@0"
I20260925 02:44:03.302990 8423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "3d6099aab36d4169978b033571a7b1b9"
format_stamp: "Formatted at 2026-09-25 02:44:03 on dist-test-slave-n563"
server_key: "2bd6d62120d7f1adeafc021bc67b9438"
server_key_iv: "18de6f7db51813d170eb0628df3b17b5"
server_key_version: "encryptionkey@0"
I20260925 02:44:03.308704 8423 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:44:03.312481 8438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:03.313918 8423 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:03.314117 8423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "3d6099aab36d4169978b033571a7b1b9"
format_stamp: "Formatted at 2026-09-25 02:44:03 on dist-test-slave-n563"
server_key: "2bd6d62120d7f1adeafc021bc67b9438"
server_key_iv: "18de6f7db51813d170eb0628df3b17b5"
server_key_version: "encryptionkey@0"
I20260925 02:44:03.314333 8423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:03.360756 8423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:03.365059 8423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:03.365491 8423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:03.384840 8423 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:43321
I20260925 02:44:03.384865 8489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:43321 every 8 connection(s)
I20260925 02:44:03.386274 8423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:03.390151 8490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:03.393153 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 8423
I20260925 02:44:03.393375 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:03.394200 5346 external_mini_cluster.cc:1428] Setting key 01fcfc0b0afddb87c0d62831ec51be12
I20260925 02:44:03.402686 8490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9: Bootstrap starting.
I20260925 02:44:03.405890 8490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:03.407018 8490 log.cc:824] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:44:03 dist-test-slave-n563 krb5kdc[8410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:03.410362 8490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9: No bootstrap required, opened a new log
I20260925 02:44:03.418609 8493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:03.396530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36692 (local address 127.5.56.190:43321)
0925 02:44:03.397361 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:03.397377 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:03.397421 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:44:03.399219 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:03.399244 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:03.399666 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:03.400113 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:03.401549 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:03.403001 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:03.404242 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:03.404947 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:03.405151 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:03.409486 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:03.409497 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:03.409517 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:03.409559 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:03.413814 (+ 4255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:03.414634 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:03.414648 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:03.414657 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:03.414894 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:03.415249 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:03.415254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:03.415257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:03.415976 (+ 719us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:03.416314 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:03.416617 (+ 303us) server_negotiation.cc:299] Negotiation successful
0925 02:44:03.417103 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":361,"threads_started":1}
I20260925 02:44:03.421360 8490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6099aab36d4169978b033571a7b1b9" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43321 } }
I20260925 02:44:03.421872 8490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:03.421985 8490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d6099aab36d4169978b033571a7b1b9, State: Initialized, Role: FOLLOWER
I20260925 02:44:03.422657 8490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6099aab36d4169978b033571a7b1b9" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43321 } }
I20260925 02:44:03.422906 8490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:03.423063 8490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:03.423251 8490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:03.425640 8490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6099aab36d4169978b033571a7b1b9" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43321 } }
I20260925 02:44:03.426182 8490 leader_election.cc:302] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d6099aab36d4169978b033571a7b1b9; no voters:
I20260925 02:44:03.426832 8490 leader_election.cc:288] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:03.426964 8495 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:03.428421 8495 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [term 1 LEADER]: Becoming Leader. State: Replica: 3d6099aab36d4169978b033571a7b1b9, State: Running, Role: LEADER
I20260925 02:44:03.429016 8495 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6099aab36d4169978b033571a7b1b9" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43321 } }
I20260925 02:44:03.429219 8490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:03.431864 8497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d6099aab36d4169978b033571a7b1b9. Latest consensus state: current_term: 1 leader_uuid: "3d6099aab36d4169978b033571a7b1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6099aab36d4169978b033571a7b1b9" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43321 } } }
I20260925 02:44:03.432303 8497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:03.432241 8496 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d6099aab36d4169978b033571a7b1b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d6099aab36d4169978b033571a7b1b9" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43321 } } }
I20260925 02:44:03.432542 8496 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:03.434465 8504 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:03.440026 8504 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:03.448833 8504 catalog_manager.cc:1347] Generated new cluster ID: 4b43f30ecd324f95a5471b2343b2c827
I20260925 02:44:03.449002 8504 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:03.480680 8504 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:03.482051 8504 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:03.494598 8504 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d6099aab36d4169978b033571a7b1b9: Generated new TSK 0
I20260925 02:44:03.495796 8504 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:44:03.547982 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43321
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:03.891256 8516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:03.891570 8516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:03.891666 8516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:03.899454 8516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:03.899586 8516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:03.899705 8516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:44:03.910107 8516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:03.911511 8516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:03.912904 8516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:03.927248 8521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:03.927332 8522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:03.927289 8524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:03.928226 8516 server_base.cc:1034] running on GCE node
I20260925 02:44:03.928937 8516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:03.930111 8516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:03.931474 8516 hybrid_clock.cc:648] HybridClock initialized: now 1790304243931389 us; error 102 us; skew 500 ppm
I20260925 02:44:03.931859 8516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:03 dist-test-slave-n563 krb5kdc[8410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304243, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:03.935393 8516 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:44:03.937137 8516 webserver.cc:460] Webserver started at http://127.5.56.129:34753/ using document root <none> and password file <none>
I20260925 02:44:03.937844 8516 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:03.937947 8516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:03.938221 8516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:03.941314 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "4ac8d885fd2648ebb128749a215046f9"
format_stamp: "Formatted at 2026-09-25 02:44:03 on dist-test-slave-n563"
server_key: "5441263edc98e5e484d92ab2f9f1f5a7"
server_key_iv: "0c910e56018071fe8afd497e5212c671"
server_key_version: "encryptionkey@0"
I20260925 02:44:03.942071 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "4ac8d885fd2648ebb128749a215046f9"
format_stamp: "Formatted at 2026-09-25 02:44:03 on dist-test-slave-n563"
server_key: "5441263edc98e5e484d92ab2f9f1f5a7"
server_key_iv: "0c910e56018071fe8afd497e5212c671"
server_key_version: "encryptionkey@0"
I20260925 02:44:03.947544 8516 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:44:03.951211 8531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:03.952769 8516 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:44:03.952922 8516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "4ac8d885fd2648ebb128749a215046f9"
format_stamp: "Formatted at 2026-09-25 02:44:03 on dist-test-slave-n563"
server_key: "5441263edc98e5e484d92ab2f9f1f5a7"
server_key_iv: "0c910e56018071fe8afd497e5212c671"
server_key_version: "encryptionkey@0"
I20260925 02:44:03.953115 8516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:03.984692 8516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:03.990823 8516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:03.991221 8516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:03.992518 8516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:03.994043 8516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:03.994138 8516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:03.994314 8516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:03.994407 8516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:04.032290 8516 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:46633
I20260925 02:44:04.032303 8645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:46633 every 8 connection(s)
I20260925 02:44:04.034276 8516 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:44:04.039300 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 8516
I20260925 02:44:04.039513 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:44:04.040143 5346 external_mini_cluster.cc:1428] Setting key 7e6b0c14f6b2cfceaef30098d3dbdf8d
Sep 25 02:44:04 dist-test-slave-n563 krb5kdc[8410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304243, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:04.056051 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:04.036973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:45267 (local address 127.5.56.190:43321)
0925 02:44:04.037446 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:04.037451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:04.038321 (+ 870us) server_negotiation.cc:407] Connection header received
0925 02:44:04.040424 (+ 2103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:04.040430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:04.040529 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:04.040652 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:04.042518 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:04.043286 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:04.044906 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:04.045331 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:04.045476 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:04.049530 (+ 4054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:04.049536 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:04.049540 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:04.049574 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:04.052416 (+ 2842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:04.053430 (+ 1014us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:04.053434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:04.053436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:04.053558 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:04.054079 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:04.054082 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:04.054084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:04.054365 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:04.054547 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:04.055566 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0925 02:44:04.055746 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":258,"threads_started":1}
I20260925 02:44:04.057875 8649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:04.037451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43321 (local address 127.5.56.129:45267)
0925 02:44:04.038162 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:04.038199 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:04.040105 (+ 1906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:04.040905 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:04.040914 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:04.041640 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:04.042312 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:04.042331 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:04.043459 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:04.043464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:04.044735 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:04.044747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:04.045585 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:04.045592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:04.045834 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:04.046574 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:04.046598 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:04.049265 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:04.052577 (+ 3312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:04.052590 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:04.052614 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:04.053301 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:04.053667 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:04.053674 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:04.053681 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:04.053949 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:04.054579 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:04.054590 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:04.055118 (+ 528us) client_negotiation.cc:769] Sending connection context
0925 02:44:04.055465 (+ 347us) client_negotiation.cc:240] Negotiation successful
0925 02:44:04.056123 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":271,"threads_started":1}
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
I20260925 02:44:04.059846 8646 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43321
I20260925 02:44:04.060206 8646 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:04.061020 8646 heartbeater.cc:511] Master 127.5.56.190:43321 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
I20260925 02:44:04.063328 8455 ts_manager.cc:194] Registered new tserver with Master: 4ac8d885fd2648ebb128749a215046f9 (127.5.56.129:46633)
I20260925 02:44:04.064815 8455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:45267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:44:04.087131 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43321
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:04.416960 8652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:04.417253 8652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:04.417348 8652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:04.424805 8652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:04.424942 8652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:04.425119 8652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:44:04.435508 8652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:04.436864 8652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:04.438244 8652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:04.451500 8660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:04.451593 8658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:04.451557 8657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:04.452621 8652 server_base.cc:1034] running on GCE node
I20260925 02:44:04.453325 8652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:04.454321 8652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:04.455582 8652 hybrid_clock.cc:648] HybridClock initialized: now 1790304244455513 us; error 66 us; skew 500 ppm
I20260925 02:44:04.455878 8652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:04 dist-test-slave-n563 krb5kdc[8410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304244, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:04.464370 8652 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:44:04.465998 8652 webserver.cc:460] Webserver started at http://127.5.56.130:40855/ using document root <none> and password file <none>
I20260925 02:44:04.466813 8652 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:04.466956 8652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:04.467290 8652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:04.470302 8652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "4712abad907a40cdb695e2f2e28b4785"
format_stamp: "Formatted at 2026-09-25 02:44:04 on dist-test-slave-n563"
server_key: "9ac813a7be5c270eef7ad1881d498ca5"
server_key_iv: "8e587286fd2b0226f3ce02b423aab223"
server_key_version: "encryptionkey@0"
I20260925 02:44:04.471407 8652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "4712abad907a40cdb695e2f2e28b4785"
format_stamp: "Formatted at 2026-09-25 02:44:04 on dist-test-slave-n563"
server_key: "9ac813a7be5c270eef7ad1881d498ca5"
server_key_iv: "8e587286fd2b0226f3ce02b423aab223"
server_key_version: "encryptionkey@0"
I20260925 02:44:04.476898 8652 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:44:04.480573 8667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:04.481995 8652 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:44:04.482199 8652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "4712abad907a40cdb695e2f2e28b4785"
format_stamp: "Formatted at 2026-09-25 02:44:04 on dist-test-slave-n563"
server_key: "9ac813a7be5c270eef7ad1881d498ca5"
server_key_iv: "8e587286fd2b0226f3ce02b423aab223"
server_key_version: "encryptionkey@0"
I20260925 02:44:04.482439 8652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:04.495973 8652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:04.500568 8652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:04.500833 8652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:04.501780 8652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:04.503309 8652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:04.503381 8652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:04.503463 8652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:04.503513 8652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:04.537604 8652 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:35537
I20260925 02:44:04.537642 8780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:35537 every 8 connection(s)
I20260925 02:44:04.539000 8652 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:44:04.547544 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 8652
I20260925 02:44:04.547739 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:44:04.548307 5346 external_mini_cluster.cc:1428] Setting key b0e2398d94760d24c550fba23763a68f
Sep 25 02:44:04 dist-test-slave-n563 krb5kdc[8410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304244, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:04.561023 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:04.541588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:47145 (local address 127.5.56.190:43321)
0925 02:44:04.541764 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:04.541769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:04.543011 (+ 1242us) server_negotiation.cc:407] Connection header received
0925 02:44:04.544762 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:04.544766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:04.544839 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:04.544928 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:04.546378 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:04.547141 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:04.549340 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:04.549842 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:04.549984 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:04.554191 (+ 4207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:04.554200 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:04.554208 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:04.554255 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:04.557499 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:04.558527 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:04.558533 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:04.558539 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:04.558716 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:04.559236 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:04.559239 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:04.559241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:04.559520 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:04.559645 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:04.560596 (+ 951us) server_negotiation.cc:299] Negotiation successful
0925 02:44:04.560780 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:44:04.562748 8783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:04.542043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43321 (local address 127.5.56.130:47145)
0925 02:44:04.542839 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:04.542883 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:04.544498 (+ 1615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:04.545135 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:04.545143 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:04.545653 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:04.546222 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:04.546240 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:04.547310 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:04.547316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:04.549185 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:04.549197 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:04.550073 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:04.550080 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:04.550383 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:04.551127 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:04.551164 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:04.553937 (+ 2773us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:04.557646 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:04.557658 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:04.557678 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:04.558359 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:04.558850 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:04.558857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:04.558864 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:04.559120 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:04.559651 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:04.559662 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:04.560158 (+ 496us) client_negotiation.cc:769] Sending connection context
0925 02:44:04.560507 (+ 349us) client_negotiation.cc:240] Negotiation successful
0925 02:44:04.561139 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":366,"threads_started":1}
I20260925 02:44:04.564255 8781 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43321
I20260925 02:44:04.564589 8781 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:04.565434 8781 heartbeater.cc:511] Master 127.5.56.190:43321 requested a full tablet report, sending...
I20260925 02:44:04.567013 8455 ts_manager.cc:194] Registered new tserver with Master: 4712abad907a40cdb695e2f2e28b4785 (127.5.56.130:35537)
I20260925 02:44:04.567833 8455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:47145
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:44:04.597484 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43321
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:04.933252 8786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:04.933579 8786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:04.933683 8786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:04.941453 8786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:04.941601 8786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:04.941728 8786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:44:04.952857 8786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43321
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:04.954301 8786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:04.955787 8786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:04.969578 8792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:04.969616 8794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:04.969555 8791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:04.970616 8786 server_base.cc:1034] running on GCE node
I20260925 02:44:04.971261 8786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:04.972201 8786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:04.973399 8786 hybrid_clock.cc:648] HybridClock initialized: now 1790304244973333 us; error 60 us; skew 500 ppm
I20260925 02:44:04.973731 8786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:04 dist-test-slave-n563 krb5kdc[8410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304244, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:04.977141 8786 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:44:04.978763 8786 webserver.cc:460] Webserver started at http://127.5.56.131:39435/ using document root <none> and password file <none>
I20260925 02:44:04.979481 8786 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:04.979633 8786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:04.979959 8786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:04.983021 8786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "7ad737ef58cd487f9b6b107981034021"
format_stamp: "Formatted at 2026-09-25 02:44:04 on dist-test-slave-n563"
server_key: "35c27138b4de9de680f1a951efb83031"
server_key_iv: "4bfaeba2dc31ccb03ebc8691a63d4a4a"
server_key_version: "encryptionkey@0"
I20260925 02:44:04.983837 8786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "7ad737ef58cd487f9b6b107981034021"
format_stamp: "Formatted at 2026-09-25 02:44:04 on dist-test-slave-n563"
server_key: "35c27138b4de9de680f1a951efb83031"
server_key_iv: "4bfaeba2dc31ccb03ebc8691a63d4a4a"
server_key_version: "encryptionkey@0"
I20260925 02:44:04.989298 8786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:44:04.992835 8801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:04.994223 8786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:44:04.994484 8786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "7ad737ef58cd487f9b6b107981034021"
format_stamp: "Formatted at 2026-09-25 02:44:04 on dist-test-slave-n563"
server_key: "35c27138b4de9de680f1a951efb83031"
server_key_iv: "4bfaeba2dc31ccb03ebc8691a63d4a4a"
server_key_version: "encryptionkey@0"
I20260925 02:44:04.994706 8786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:05.009689 8786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:05.014205 8786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:05.014549 8786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:05.015578 8786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:05.016942 8786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:05.017052 8786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:05.017213 8786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:05.017292 8786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:05.053923 8786 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:39513
I20260925 02:44:05.053982 8915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:39513 every 8 connection(s)
I20260925 02:44:05.055389 8786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:44:05.057508 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 8786
I20260925 02:44:05.057711 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:44:05.058259 5346 external_mini_cluster.cc:1428] Setting key 1fe85b129ef4b7ccaadb837bc5921a1b
I20260925 02:44:05.067998 8646 heartbeater.cc:503] Master 127.5.56.190:43321 was elected leader, sending a full tablet report...
Sep 25 02:44:05 dist-test-slave-n563 krb5kdc[8410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304244, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:05.077775 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:05.058448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:53175 (local address 127.5.56.190:43321)
0925 02:44:05.058680 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:05.058686 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:05.060136 (+ 1450us) server_negotiation.cc:407] Connection header received
0925 02:44:05.062282 (+ 2146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:05.062287 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:05.062372 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:05.062486 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:05.063990 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.065379 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.067049 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.067488 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.067630 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:05.071683 (+ 4053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:05.071688 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:05.071691 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:05.071722 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:05.074628 (+ 2906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:05.075389 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:05.075399 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:05.075402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:05.075539 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:05.075998 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:05.076002 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:05.076004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:05.076275 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:05.076450 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:05.077164 (+ 714us) server_negotiation.cc:299] Negotiation successful
0925 02:44:05.077401 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:44:05.078887 8918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:05.059132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43321 (local address 127.5.56.131:53175)
0925 02:44:05.059981 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:05.060020 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:05.062014 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:05.062723 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:05.062732 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:05.063239 (+ 507us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:05.063819 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:05.063836 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.065536 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.065541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:05.066896 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:05.066909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.067729 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.067759 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:05.067989 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:05.068744 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:05.068775 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:05.071454 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:05.074782 (+ 3328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:05.074792 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:05.074812 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:05.075255 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:05.075654 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:05.075659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:05.075663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:05.075859 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:05.076414 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:05.076423 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:05.076796 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 02:44:05.077069 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 02:44:05.077649 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":386,"threads_started":1}
I20260925 02:44:05.080360 8916 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43321
I20260925 02:44:05.080690 8916 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:05.081444 8916 heartbeater.cc:511] Master 127.5.56.190:43321 requested a full tablet report, sending...
I20260925 02:44:05.082922 8455 ts_manager.cc:194] Registered new tserver with Master: 7ad737ef58cd487f9b6b107981034021 (127.5.56.131:39513)
I20260925 02:44:05.083722 8455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:53175
I20260925 02:44:05.086366 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:44:05 dist-test-slave-n563 krb5kdc[8410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304245, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 02:44:05 dist-test-slave-n563 krb5kdc[8410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304245, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:05.137077 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:05.122545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36700 (local address 127.5.56.190:43321)
0925 02:44:05.122751 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:05.122760 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:05.123261 (+ 501us) server_negotiation.cc:407] Connection header received
0925 02:44:05.123500 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:05.123504 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:05.123571 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:05.123681 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:05.125082 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.125826 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.127177 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.127599 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.127736 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:05.131843 (+ 4107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:05.131849 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:05.131852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:05.131888 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:05.134466 (+ 2578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:05.135097 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:05.135104 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:05.135113 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:05.135249 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:05.135648 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:05.135652 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:05.135654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:05.135970 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:05.136224 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:05.136528 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 02:44:05.136749 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260925 02:44:05.137642 8455 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36700
I20260925 02:44:05.139317 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8516
Sep 25 02:44:05 dist-test-slave-n563 krb5kdc[8410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304245, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:44:05.155201 8927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:05.140316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58440 (local address 127.5.56.129:46633)
0925 02:44:05.140926 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:05.140933 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:05.140956 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:44:05.141013 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:05.141020 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:05.141229 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:05.141376 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:05.142607 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.143629 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.144882 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.145457 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.145552 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:05.149543 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:05.149557 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:05.149573 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:05.149625 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:05.152432 (+ 2807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:05.153105 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:05.153112 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:05.153114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:05.153235 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:05.153672 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:05.153675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:05.153677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:05.154017 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:05.154264 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:05.154582 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 02:44:05.154781 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":296,"threads_started":1}
W20260925 02:44:05.156528 8641 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:58440
W20260925 02:44:05.157775 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8516: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:44:05.158049 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 8516
I20260925 02:44:05.175354 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8652
Sep 25 02:44:05 dist-test-slave-n563 krb5kdc[8410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304245, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:44:05.191547 8928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:05.176280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60450 (local address 127.5.56.130:35537)
0925 02:44:05.176844 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:05.176851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:05.176878 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:44:05.176940 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:05.176946 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:05.177168 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:05.177299 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:05.178630 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.179656 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.180856 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.181401 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.181499 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:05.185701 (+ 4202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:05.185713 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:05.185729 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:05.185767 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:05.188971 (+ 3204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:05.189648 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:05.189656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:05.189658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:05.189783 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:05.190125 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:05.190128 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:05.190130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:05.190542 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:05.190791 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:05.191047 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 02:44:05.191258 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":302,"threads_started":1}
W20260925 02:44:05.192836 8776 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60450
W20260925 02:44:05.193972 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8652: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:44:05.194139 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 8652
I20260925 02:44:05.211045 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8786
Sep 25 02:44:05 dist-test-slave-n563 krb5kdc[8410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304245, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:44:05.228871 8929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:05.211975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.5.56.131:39513)
0925 02:44:05.212730 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:05.212737 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:05.212767 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:44:05.212828 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:05.212834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:05.213049 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:05.213189 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:05.214718 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.215768 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.217006 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:05.217642 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:05.217742 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:05.222179 (+ 4437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:05.222192 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:05.222208 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:05.222244 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:05.225138 (+ 2894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:05.225980 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:05.225988 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:05.225990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:05.226191 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:05.226691 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:05.226696 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:05.226699 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:05.227284 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:05.227673 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:05.228043 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 02:44:05.228393 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":415,"threads_started":1}
W20260925 02:44:05.230762 8911 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48038
W20260925 02:44:05.232157 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8786: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:44:05.232365 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 8786
I20260925 02:44:05.250005 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8423
I20260925 02:44:05.251340 8485 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:05.358103 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 8423
2026-09-25T02:44:05Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3739 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:44:05 dist-test-slave-n563 krb5kdc[8933](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:44:05 dist-test-slave-n563 krb5kdc[8933](info): set up 2 sockets
Sep 25 02:44:05 dist-test-slave-n563 krb5kdc[8933](info): commencing operation
krb5kdc: starting...
W20260925 02:44:06.464269 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:44:06 dist-test-slave-n563 krb5kdc[8933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:44:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:44:06.677512 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:36817
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:36817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:07.016502 8946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:07.016835 8946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:07.016937 8946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:07.024554 8946 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:07.024691 8946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:07.024740 8946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:07.024799 8946 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:07.024844 8946 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:07.035251 8946 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:36817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:36817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:07.036691 8946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:07.038085 8946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:07.050678 8955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:07.050629 8953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:07.050721 8952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:07.051594 8946 server_base.cc:1034] running on GCE node
I20260925 02:44:07.052416 8946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:07.053934 8946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:07.055168 8946 hybrid_clock.cc:648] HybridClock initialized: now 1790304247055114 us; error 46 us; skew 500 ppm
I20260925 02:44:07.055533 8946 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:07 dist-test-slave-n563 krb5kdc[8933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304247, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:07.059048 8946 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:44:07.060613 8946 webserver.cc:460] Webserver started at http://127.5.56.190:44405/ using document root <none> and password file <none>
I20260925 02:44:07.061305 8946 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:07.061409 8946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:07.061688 8946 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:07.064867 8946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "482e51b8629a417cbf2ffd78fcf16ada"
format_stamp: "Formatted at 2026-09-25 02:44:07 on dist-test-slave-n563"
server_key: "d3a1345f11fb245cad062044a5f3253e"
server_key_iv: "3f9125734be51c131085719beb66a597"
server_key_version: "encryptionkey@0"
I20260925 02:44:07.065631 8946 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "482e51b8629a417cbf2ffd78fcf16ada"
format_stamp: "Formatted at 2026-09-25 02:44:07 on dist-test-slave-n563"
server_key: "d3a1345f11fb245cad062044a5f3253e"
server_key_iv: "3f9125734be51c131085719beb66a597"
server_key_version: "encryptionkey@0"
I20260925 02:44:07.071137 8946 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:44:07.074692 8962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:07.076126 8946 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260925 02:44:07.076314 8946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "482e51b8629a417cbf2ffd78fcf16ada"
format_stamp: "Formatted at 2026-09-25 02:44:07 on dist-test-slave-n563"
server_key: "d3a1345f11fb245cad062044a5f3253e"
server_key_iv: "3f9125734be51c131085719beb66a597"
server_key_version: "encryptionkey@0"
I20260925 02:44:07.076521 8946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:07.098242 8946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:07.103045 8946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:07.103366 8946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:07.122962 8946 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:36817
I20260925 02:44:07.123018 9013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:36817 every 8 connection(s)
I20260925 02:44:07.124413 8946 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:07.128329 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 8946
I20260925 02:44:07.128427 9014 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:07.128628 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:07.129282 5346 external_mini_cluster.cc:1428] Setting key f98b1e753bd10e76872c0a6e8fd90f14
I20260925 02:44:07.143208 9014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada: Bootstrap starting.
Sep 25 02:44:07 dist-test-slave-n563 krb5kdc[8933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:07.147164 9014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:07.148439 9014 log.cc:824] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:07.151520 9014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada: No bootstrap required, opened a new log
I20260925 02:44:07.153988 9017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:07.131501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38190 (local address 127.5.56.190:36817)
0925 02:44:07.132341 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:07.132354 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:07.132401 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:44:07.134628 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:07.134655 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:07.135133 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:07.135498 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:07.137476 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:07.138894 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:07.140818 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:07.141552 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:07.141705 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:07.146508 (+ 4803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:07.146518 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:07.146536 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:07.146579 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:07.149755 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:07.150534 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:07.150541 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:07.150548 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:07.150725 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:07.151136 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:07.151140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:07.151142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:07.151726 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:07.152051 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:07.152503 (+ 452us) server_negotiation.cc:299] Negotiation successful
0925 02:44:07.152941 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":395,"threads_started":1}
I20260925 02:44:07.159816 9014 raft_consensus.cc:348] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482e51b8629a417cbf2ffd78fcf16ada" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36817 } }
I20260925 02:44:07.160229 9014 raft_consensus.cc:374] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:07.160354 9014 raft_consensus.cc:729] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482e51b8629a417cbf2ffd78fcf16ada, State: Initialized, Role: FOLLOWER
I20260925 02:44:07.161021 9014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482e51b8629a417cbf2ffd78fcf16ada" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36817 } }
I20260925 02:44:07.161291 9014 raft_consensus.cc:388] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:07.161425 9014 raft_consensus.cc:482] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:07.161612 9014 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:07.164391 9014 raft_consensus.cc:504] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482e51b8629a417cbf2ffd78fcf16ada" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36817 } }
I20260925 02:44:07.165019 9014 leader_election.cc:302] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 482e51b8629a417cbf2ffd78fcf16ada; no voters:
I20260925 02:44:07.165637 9014 leader_election.cc:288] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:07.165777 9019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:07.166555 9019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [term 1 LEADER]: Becoming Leader. State: Replica: 482e51b8629a417cbf2ffd78fcf16ada, State: Running, Role: LEADER
I20260925 02:44:07.167153 9019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482e51b8629a417cbf2ffd78fcf16ada" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36817 } }
I20260925 02:44:07.168061 9014 sys_catalog.cc:564] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:07.170219 9020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "482e51b8629a417cbf2ffd78fcf16ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482e51b8629a417cbf2ffd78fcf16ada" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36817 } } }
I20260925 02:44:07.170499 9021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader 482e51b8629a417cbf2ffd78fcf16ada. Latest consensus state: current_term: 1 leader_uuid: "482e51b8629a417cbf2ffd78fcf16ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "482e51b8629a417cbf2ffd78fcf16ada" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36817 } } }
I20260925 02:44:07.170780 9020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:07.170847 9021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:07.173130 9024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:07.179175 9024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:07.189576 9024 catalog_manager.cc:1347] Generated new cluster ID: d3e0fd84b651475fb7d638d4fef6a592
I20260925 02:44:07.189741 9024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:07.204434 9024 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:07.205901 9024 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:07.219074 9024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 482e51b8629a417cbf2ffd78fcf16ada: Generated new TSK 0
I20260925 02:44:07.220027 9024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:44:07.274933 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36817
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:07.637903 9041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:07.638192 9041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:07.638370 9041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:07.645973 9041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:07.646088 9041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:07.646301 9041 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:44:07.656499 9041 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:07.657879 9041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:07.659363 9041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:07.673305 9047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:07.673563 9049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:07.673914 9046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:07.674729 9041 server_base.cc:1034] running on GCE node
I20260925 02:44:07.675422 9041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:07.676385 9041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:07.677582 9041 hybrid_clock.cc:648] HybridClock initialized: now 1790304247677517 us; error 56 us; skew 500 ppm
I20260925 02:44:07.677906 9041 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:07 dist-test-slave-n563 krb5kdc[8933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304247, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:07.681331 9041 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:44:07.682945 9041 webserver.cc:460] Webserver started at http://127.5.56.129:46767/ using document root <none> and password file <none>
I20260925 02:44:07.683620 9041 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:07.683720 9041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:07.684001 9041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:07.687121 9041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "8469d89b243e4ec78c4a8a1702db1dcd"
format_stamp: "Formatted at 2026-09-25 02:44:07 on dist-test-slave-n563"
server_key: "e234124762d52f49753ff4fe528ffd9a"
server_key_iv: "398ced2ffa26982404b312264cf2b45c"
server_key_version: "encryptionkey@0"
I20260925 02:44:07.687893 9041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "8469d89b243e4ec78c4a8a1702db1dcd"
format_stamp: "Formatted at 2026-09-25 02:44:07 on dist-test-slave-n563"
server_key: "e234124762d52f49753ff4fe528ffd9a"
server_key_iv: "398ced2ffa26982404b312264cf2b45c"
server_key_version: "encryptionkey@0"
I20260925 02:44:07.693305 9041 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 02:44:07.696857 9056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:07.698273 9041 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:44:07.698487 9041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "8469d89b243e4ec78c4a8a1702db1dcd"
format_stamp: "Formatted at 2026-09-25 02:44:07 on dist-test-slave-n563"
server_key: "e234124762d52f49753ff4fe528ffd9a"
server_key_iv: "398ced2ffa26982404b312264cf2b45c"
server_key_version: "encryptionkey@0"
I20260925 02:44:07.698721 9041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:07.719841 9041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:07.724309 9041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:07.724618 9041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:07.725658 9041 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:07.727138 9041 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:07.727246 9041 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:07.727396 9041 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:07.727471 9041 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:07.762281 9041 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:43781
I20260925 02:44:07.762375 9169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:43781 every 8 connection(s)
I20260925 02:44:07.763738 9041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:44:07.769754 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 9041
I20260925 02:44:07.770004 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:44:07.770919 5346 external_mini_cluster.cc:1428] Setting key c81e386d48ff05635f15ded478a5d7b0
Sep 25 02:44:07 dist-test-slave-n563 krb5kdc[8933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304247, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:07.788990 9172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:07.766597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:53469 (local address 127.5.56.190:36817)
0925 02:44:07.767096 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:07.767100 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:07.768018 (+ 918us) server_negotiation.cc:407] Connection header received
0925 02:44:07.770511 (+ 2493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:07.770516 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:07.770606 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:07.770704 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:07.772839 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:07.773759 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:07.776434 (+ 2675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:07.776853 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:07.776956 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:07.781491 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:07.781501 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:07.781509 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:07.781558 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:07.785248 (+ 3690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:07.786126 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:07.786136 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:07.786142 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:07.786300 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:07.786822 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:07.786831 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:07.786836 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:07.787267 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:07.787424 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:07.788218 (+ 794us) server_negotiation.cc:299] Negotiation successful
0925 02:44:07.788499 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":262,"threads_started":1}
I20260925 02:44:07.789769 9173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:07.767034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36817 (local address 127.5.56.129:53469)
0925 02:44:07.767821 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:07.767881 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:07.770126 (+ 2245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:07.771028 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:07.771041 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:07.771766 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:07.772603 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:07.772642 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:07.773925 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:07.773934 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:07.776273 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:07.776294 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:07.777105 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:07.777114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:07.777483 (+ 369us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:07.778264 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:07.778302 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:07.781243 (+ 2941us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:07.785442 (+ 4199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:07.785450 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:07.785468 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:07.785958 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:07.786480 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:07.786484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:07.786486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:07.786687 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:07.787393 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:07.787400 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:07.787781 (+ 381us) client_negotiation.cc:769] Sending connection context
0925 02:44:07.788032 (+ 251us) client_negotiation.cc:240] Negotiation successful
0925 02:44:07.788477 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":297,"threads_started":1}
I20260925 02:44:07.791512 9170 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36817
I20260925 02:44:07.791846 9170 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:07.792608 9170 heartbeater.cc:511] Master 127.5.56.190:36817 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
I20260925 02:44:07.794900 8979 ts_manager.cc:194] Registered new tserver with Master: 8469d89b243e4ec78c4a8a1702db1dcd (127.5.56.129:43781)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
I20260925 02:44:07.796622 8979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:53469
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:44:07.821696 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36817
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:08.158022 9176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:08.158387 9176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:08.158582 9176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:08.166261 9176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:08.166473 9176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:08.166658 9176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:44:08.177453 9176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:08.178948 9176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:08.180485 9176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:08.194007 9181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:08.193972 9182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:08.194168 9184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:08.195034 9176 server_base.cc:1034] running on GCE node
I20260925 02:44:08.195636 9176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:08.196586 9176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:08.197785 9176 hybrid_clock.cc:648] HybridClock initialized: now 1790304248197714 us; error 64 us; skew 500 ppm
I20260925 02:44:08.198119 9176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:08 dist-test-slave-n563 krb5kdc[8933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:08.201413 9176 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:44:08.203086 9176 webserver.cc:460] Webserver started at http://127.5.56.130:35427/ using document root <none> and password file <none>
I20260925 02:44:08.203886 9176 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:08.204041 9176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:08.204360 9176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:08.207518 9176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "4b392e1e8f0e4eac9c2e8bbf6713deb4"
format_stamp: "Formatted at 2026-09-25 02:44:08 on dist-test-slave-n563"
server_key: "60a7b848e3bcf7d9f2f9f55dcdf87c02"
server_key_iv: "cf7e4605a628e8d8fca05b37247dadfc"
server_key_version: "encryptionkey@0"
I20260925 02:44:08.208323 9176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "4b392e1e8f0e4eac9c2e8bbf6713deb4"
format_stamp: "Formatted at 2026-09-25 02:44:08 on dist-test-slave-n563"
server_key: "60a7b848e3bcf7d9f2f9f55dcdf87c02"
server_key_iv: "cf7e4605a628e8d8fca05b37247dadfc"
server_key_version: "encryptionkey@0"
I20260925 02:44:08.213846 9176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:44:08.217636 9191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:08.219103 9176 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:44:08.219280 9176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "4b392e1e8f0e4eac9c2e8bbf6713deb4"
format_stamp: "Formatted at 2026-09-25 02:44:08 on dist-test-slave-n563"
server_key: "60a7b848e3bcf7d9f2f9f55dcdf87c02"
server_key_iv: "cf7e4605a628e8d8fca05b37247dadfc"
server_key_version: "encryptionkey@0"
I20260925 02:44:08.219496 9176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:08.246696 9176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:08.250922 9176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:08.251235 9176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:08.252341 9176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:08.253796 9176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:08.253897 9176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:08.254046 9176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:08.254117 9176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:08.288825 9176 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:37877
I20260925 02:44:08.288879 9303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:37877 every 8 connection(s)
I20260925 02:44:08.290318 9176 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:44:08.292640 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 9176
I20260925 02:44:08.292837 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:44:08.293390 5346 external_mini_cluster.cc:1428] Setting key 4a8d9262c996ddf3d8d3df77e7d25628
Sep 25 02:44:08 dist-test-slave-n563 krb5kdc[8933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:08.314311 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:08.293447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:46165 (local address 127.5.56.190:36817)
0925 02:44:08.293924 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:08.293933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:08.295217 (+ 1284us) server_negotiation.cc:407] Connection header received
0925 02:44:08.296999 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:08.297004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:08.297098 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:08.297219 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:08.298986 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.300248 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.302412 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.302927 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.303032 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:08.306866 (+ 3834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:08.306875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:08.306884 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:08.306934 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:08.310722 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:08.311523 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:08.311531 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:08.311538 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:08.311716 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:08.312183 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:08.312191 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:08.312197 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:08.312663 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:08.312843 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:08.313595 (+ 752us) server_negotiation.cc:299] Negotiation successful
0925 02:44:08.313864 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":225,"threads_started":1}
I20260925 02:44:08.315099 9308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:08.294031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36817 (local address 127.5.56.130:46165)
0925 02:44:08.294903 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:08.294959 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:08.296710 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:08.297465 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:08.297473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:08.297989 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:08.298780 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:08.298813 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.300539 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.300548 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:08.302217 (+ 1669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:08.302231 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.303133 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.303139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:08.303344 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:08.304102 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:08.304127 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:08.306608 (+ 2481us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:08.310888 (+ 4280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:08.310896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:08.310911 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:08.311377 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:08.311831 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:08.311835 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:08.311837 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:08.312052 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:08.312806 (+ 754us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:08.312812 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:08.313192 (+ 380us) client_negotiation.cc:769] Sending connection context
0925 02:44:08.313424 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 02:44:08.313852 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":389,"threads_started":1}
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
I20260925 02:44:08.316684 9304 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36817
I20260925 02:44:08.317013 9304 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:08.317767 9304 heartbeater.cc:511] Master 127.5.56.190:36817 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
I20260925 02:44:08.319567 8979 ts_manager.cc:194] Registered new tserver with Master: 4b392e1e8f0e4eac9c2e8bbf6713deb4 (127.5.56.130:37877)
I20260925 02:44:08.320858 8979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:46165
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:44:08.344256 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36817
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:08.676855 9311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:08.677152 9311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:08.677349 9311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:08.684878 9311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:08.685002 9311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:08.685194 9311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:44:08.695626 9311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:08.697269 9311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:08.698743 9311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:08.712646 9316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:08.712723 9317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:08.712723 9319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:08.713681 9311 server_base.cc:1034] running on GCE node
I20260925 02:44:08.714327 9311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:08.715349 9311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:08.716621 9311 hybrid_clock.cc:648] HybridClock initialized: now 1790304248716577 us; error 54 us; skew 500 ppm
I20260925 02:44:08.716976 9311 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:08 dist-test-slave-n563 krb5kdc[8933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:08.720165 9311 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:44:08.721750 9311 webserver.cc:460] Webserver started at http://127.5.56.131:38299/ using document root <none> and password file <none>
I20260925 02:44:08.722539 9311 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:08.722697 9311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:08.723083 9311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:08.726205 9311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "cfdc36ac3fb440898e5fd5f13d74d7c7"
format_stamp: "Formatted at 2026-09-25 02:44:08 on dist-test-slave-n563"
server_key: "1ff1ff20abe9e592de93380f0e811da6"
server_key_iv: "d1c1e3e6b624a68b44617b886c6e7f10"
server_key_version: "encryptionkey@0"
I20260925 02:44:08.727082 9311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "cfdc36ac3fb440898e5fd5f13d74d7c7"
format_stamp: "Formatted at 2026-09-25 02:44:08 on dist-test-slave-n563"
server_key: "1ff1ff20abe9e592de93380f0e811da6"
server_key_iv: "d1c1e3e6b624a68b44617b886c6e7f10"
server_key_version: "encryptionkey@0"
I20260925 02:44:08.732620 9311 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:44:08.736439 9326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:08.737874 9311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:08.738075 9311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "cfdc36ac3fb440898e5fd5f13d74d7c7"
format_stamp: "Formatted at 2026-09-25 02:44:08 on dist-test-slave-n563"
server_key: "1ff1ff20abe9e592de93380f0e811da6"
server_key_iv: "d1c1e3e6b624a68b44617b886c6e7f10"
server_key_version: "encryptionkey@0"
I20260925 02:44:08.738315 9311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:08.756457 9311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:08.761190 9311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:08.761505 9311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:08.762802 9311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:08.764633 9311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:08.764734 9311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:08.764868 9311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:08.764933 9311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:08.799780 9311 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:38077
I20260925 02:44:08.799816 9438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:38077 every 8 connection(s)
I20260925 02:44:08.800099 9170 heartbeater.cc:503] Master 127.5.56.190:36817 was elected leader, sending a full tablet report...
I20260925 02:44:08.801474 9311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:44:08.804776 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 9311
I20260925 02:44:08.805051 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:44:08.805595 5346 external_mini_cluster.cc:1428] Setting key 35dbd50a81c3cfb8f4b9122524ab378c
Sep 25 02:44:08 dist-test-slave-n563 krb5kdc[8933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304248, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:08.822674 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:08.804524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:41595 (local address 127.5.56.190:36817)
0925 02:44:08.804762 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:08.804771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:08.806171 (+ 1400us) server_negotiation.cc:407] Connection header received
0925 02:44:08.808293 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:08.808298 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:08.808386 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:08.808500 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:08.810200 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.811067 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.812649 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.813061 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.813195 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:08.816913 (+ 3718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:08.816918 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:08.816921 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:08.816957 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:08.819546 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:08.820295 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:08.820302 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:08.820306 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:08.820433 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:08.820881 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:08.820887 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:08.820890 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:08.821187 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:08.821350 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:08.822073 (+ 723us) server_negotiation.cc:299] Negotiation successful
0925 02:44:08.822306 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 02:44:08.823889 9441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:08.805018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36817 (local address 127.5.56.131:41595)
0925 02:44:08.805972 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:08.806036 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:08.808021 (+ 1985us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:08.808797 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:08.808811 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:08.809376 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:08.810016 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:08.810034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.811229 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.811235 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:08.812500 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:08.812512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.813304 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.813312 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:08.813572 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:08.814295 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:08.814318 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:08.816666 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:08.819695 (+ 3029us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:08.819705 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:08.819723 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:08.820173 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:08.820555 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:08.820560 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:08.820564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:08.820766 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:08.821319 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:08.821327 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:08.821703 (+ 376us) client_negotiation.cc:769] Sending connection context
0925 02:44:08.821978 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 02:44:08.822581 (+ 603us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":339,"threads_started":1}
I20260925 02:44:08.825278 9439 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36817
I20260925 02:44:08.825606 9439 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:08.826362 9439 heartbeater.cc:511] Master 127.5.56.190:36817 requested a full tablet report, sending...
I20260925 02:44:08.827823 8979 ts_manager.cc:194] Registered new tserver with Master: cfdc36ac3fb440898e5fd5f13d74d7c7 (127.5.56.131:38077)
I20260925 02:44:08.828594 8979 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:41595
I20260925 02:44:08.833926 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:44:08 dist-test-slave-n563 krb5kdc[8933](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:44:08 dist-test-slave-n563 krb5kdc[8933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:44:08.887066 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:08.870181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47788 (local address 127.5.56.129:43781)
0925 02:44:08.870873 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:08.870880 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:08.870908 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:44:08.870972 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:08.870981 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:08.871202 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:08.871356 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:08.872750 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.874206 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.875560 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.876221 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.876335 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:08.880695 (+ 4360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:08.880715 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:08.880732 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:08.880802 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:08.884170 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:08.884960 (+ 790us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:08.884969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:08.884971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:08.885123 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:08.885487 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:08.885491 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:08.885493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:08.885907 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:08.886168 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:08.886466 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 02:44:08.886703 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":388,"threads_started":1}
Sep 25 02:44:08 dist-test-slave-n563 krb5kdc[8933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:08.907056 9307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:08.891911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.5.56.190:36817)
0925 02:44:08.892131 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:08.892138 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:08.892285 (+ 147us) server_negotiation.cc:407] Connection header received
0925 02:44:08.892503 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:08.892508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:08.892591 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:08.892700 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:08.893979 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.894799 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.896095 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.896540 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.896680 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:08.901314 (+ 4634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:08.901319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:08.901323 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:08.901362 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:08.904164 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:08.904902 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:08.904908 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:08.904911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:08.905058 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:08.905451 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:08.905456 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:08.905459 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:08.905813 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:08.906114 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:08.906472 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 02:44:08.906711 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 02:44:08.907696 8979 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36226
I20260925 02:44:08.908761 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 9041
I20260925 02:44:08.922477 9446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:08.909793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47800 (local address 127.5.56.129:43781)
0925 02:44:08.910025 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:08.910032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:08.910174 (+ 142us) server_negotiation.cc:407] Connection header received
0925 02:44:08.910379 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:08.910413 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:08.910530 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:08.910639 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:08.911862 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.912572 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.913970 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:08.914511 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:08.914661 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:08.916996 (+ 2335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:08.917001 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:08.917005 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:08.917040 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:08.919627 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:08.920309 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:08.920316 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:08.920319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:08.920461 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:08.920849 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:08.920854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:08.920857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:08.921128 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:08.921374 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:08.921708 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 02:44:08.921953 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:44:08.923202 9165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:09.009442 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 9041
I20260925 02:44:09.027011 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 9176
Sep 25 02:44:09 dist-test-slave-n563 krb5kdc[8933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:44:09.044659 9452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:09.027936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47298 (local address 127.5.56.130:37877)
0925 02:44:09.028485 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:09.028492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:09.028518 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:44:09.028580 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:09.028587 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:09.028801 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:09.028932 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:09.030379 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:09.031459 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:09.032784 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:09.033360 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:09.033538 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:09.038499 (+ 4961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:09.038514 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:09.038531 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:09.038573 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:09.041641 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:09.042523 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:09.042535 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:09.042538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:09.042727 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:09.043097 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:09.043100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:09.043102 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:09.043511 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:09.043762 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:09.044032 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 02:44:09.044294 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":282,"threads_started":1}
I20260925 02:44:09.046013 9299 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:09.131158 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 9176
I20260925 02:44:09.148432 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 9311
Sep 25 02:44:09 dist-test-slave-n563 krb5kdc[8933](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:44:09.165947 9454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:09.149507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53286 (local address 127.5.56.131:38077)
0925 02:44:09.150243 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:09.150250 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:09.150281 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:44:09.150342 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:09.150348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:09.150657 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:09.150801 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:09.152100 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:09.153186 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:09.154618 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:09.155193 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:09.155295 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:09.159834 (+ 4539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:09.159850 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:09.159867 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:09.159911 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:09.163025 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:09.163941 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:09.163949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:09.163951 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:09.164100 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:09.164440 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:09.164443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:09.164445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:09.164819 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:09.165065 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:09.165321 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 02:44:09.165545 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":420,"threads_started":1}
I20260925 02:44:09.167644 9434 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:09.257205 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 9311
I20260925 02:44:09.275293 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 8946
I20260925 02:44:09.276594 9009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:09.385105 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 8946
2026-09-25T02:44:09Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4028 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:44:09 dist-test-slave-n563 krb5kdc[9459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:44:09 dist-test-slave-n563 krb5kdc[9459](info): set up 2 sockets
Sep 25 02:44:09 dist-test-slave-n563 krb5kdc[9459](info): commencing operation
krb5kdc: starting...
W20260925 02:44:10.484351 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:44:10 dist-test-slave-n563 krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:44:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:44:10.699313 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:46667
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:46667
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:11.032980 9473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:11.033368 9473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:11.033561 9473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:11.040999 9473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:11.041165 9473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:11.041252 9473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:11.041335 9473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:11.041407 9473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:11.051609 9473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:46667
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:46667
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:11.053049 9473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:11.054459 9473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:11.066664 9481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:11.066704 9479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:11.066612 9478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:11.067404 9473 server_base.cc:1034] running on GCE node
I20260925 02:44:11.068372 9473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:11.069985 9473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:11.071192 9473 hybrid_clock.cc:648] HybridClock initialized: now 1790304251071128 us; error 54 us; skew 500 ppm
I20260925 02:44:11.071594 9473 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:11 dist-test-slave-n563 krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304251, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:11.074826 9473 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:44:11.075984 9473 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:11.076083 9473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:11.076356 9473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:11.079463 9473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "9163b4667ce34676b134d590abd6fb5b"
format_stamp: "Formatted at 2026-09-25 02:44:11 on dist-test-slave-n563"
server_key: "1900a51880bdbdcd432349a26b6c7f46"
server_key_iv: "3be8e75d2211e3a40d0598a08735f970"
server_key_version: "encryptionkey@0"
I20260925 02:44:11.080245 9473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "9163b4667ce34676b134d590abd6fb5b"
format_stamp: "Formatted at 2026-09-25 02:44:11 on dist-test-slave-n563"
server_key: "1900a51880bdbdcd432349a26b6c7f46"
server_key_iv: "3be8e75d2211e3a40d0598a08735f970"
server_key_version: "encryptionkey@0"
I20260925 02:44:11.085582 9473 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:44:11.089473 9487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:11.091028 9473 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:44:11.091219 9473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "9163b4667ce34676b134d590abd6fb5b"
format_stamp: "Formatted at 2026-09-25 02:44:11 on dist-test-slave-n563"
server_key: "1900a51880bdbdcd432349a26b6c7f46"
server_key_iv: "3be8e75d2211e3a40d0598a08735f970"
server_key_version: "encryptionkey@0"
I20260925 02:44:11.091457 9473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:11.119433 9473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:11.123993 9473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:11.124297 9473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:11.144021 9473 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:46667
I20260925 02:44:11.144039 9538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:46667 every 8 connection(s)
I20260925 02:44:11.145030 9473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:11.148937 9539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:11.150032 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 9473
I20260925 02:44:11.150292 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:11.151117 5346 external_mini_cluster.cc:1428] Setting key 332a8f32aa9797e7690963884146556c
I20260925 02:44:11.164206 9539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b: Bootstrap starting.
Sep 25 02:44:11 dist-test-slave-n563 krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:11.168028 9539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:11.169414 9539 log.cc:824] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:11.172806 9539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b: No bootstrap required, opened a new log
I20260925 02:44:11.176031 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:11.153295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36188 (local address 127.5.56.190:46667)
0925 02:44:11.154156 (+ 861us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:11.154168 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:11.154209 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:44:11.155901 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:11.155925 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:11.156366 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:11.156757 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:11.158716 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:11.160070 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:11.161988 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:11.162803 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:11.162947 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:11.167648 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:11.167656 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:11.167673 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:11.167714 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:11.171182 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:11.171960 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:11.171969 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:11.171977 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:11.172198 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:11.172585 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:11.172590 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:11.172593 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:11.173346 (+ 753us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:11.173675 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:11.174016 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 02:44:11.174501 (+ 485us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":404,"threads_started":1}
I20260925 02:44:11.181964 9539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9163b4667ce34676b134d590abd6fb5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46667 } }
I20260925 02:44:11.182324 9539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:11.182472 9539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9163b4667ce34676b134d590abd6fb5b, State: Initialized, Role: FOLLOWER
I20260925 02:44:11.183063 9539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9163b4667ce34676b134d590abd6fb5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46667 } }
I20260925 02:44:11.183262 9539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:11.183362 9539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:11.183477 9539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:11.185833 9539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9163b4667ce34676b134d590abd6fb5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46667 } }
I20260925 02:44:11.186292 9539 leader_election.cc:302] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9163b4667ce34676b134d590abd6fb5b; no voters:
I20260925 02:44:11.186939 9539 leader_election.cc:288] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:11.187126 9544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:11.187800 9544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [term 1 LEADER]: Becoming Leader. State: Replica: 9163b4667ce34676b134d590abd6fb5b, State: Running, Role: LEADER
I20260925 02:44:11.188427 9544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9163b4667ce34676b134d590abd6fb5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46667 } }
I20260925 02:44:11.189350 9539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:11.191649 9546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9163b4667ce34676b134d590abd6fb5b. Latest consensus state: current_term: 1 leader_uuid: "9163b4667ce34676b134d590abd6fb5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9163b4667ce34676b134d590abd6fb5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46667 } } }
I20260925 02:44:11.192028 9546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:11.192044 9545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9163b4667ce34676b134d590abd6fb5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9163b4667ce34676b134d590abd6fb5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 46667 } } }
I20260925 02:44:11.192752 9545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:11.195062 9553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:11.200631 9553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:11.209632 9553 catalog_manager.cc:1347] Generated new cluster ID: 09706dd30a804a868e895b3389cb2107
I20260925 02:44:11.209758 9553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:11.244191 9553 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:11.246253 9553 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:11.257591 9553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b: Generated new TSK 0
I20260925 02:44:11.258617 9553 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:44:11.322861 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:46667
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:11.685150 9565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:11.685441 9565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:11.685606 9565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:11.693346 9565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:11.693480 9565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:11.693714 9565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:44:11.704244 9565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:46667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:11.705613 9565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:11.707053 9565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:11.720569 9573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:11.720613 9570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:11.720623 9571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:11.721503 9565 server_base.cc:1034] running on GCE node
I20260925 02:44:11.722203 9565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:11.723285 9565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:11.724516 9565 hybrid_clock.cc:648] HybridClock initialized: now 1790304251724461 us; error 50 us; skew 500 ppm
I20260925 02:44:11.724879 9565 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:11 dist-test-slave-n563 krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304251, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:11.728351 9565 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:44:11.729718 9565 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:11.729861 9565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:11.730199 9565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:11.733376 9565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "5ba55afe232b45889dd3fc45f35d5f08"
format_stamp: "Formatted at 2026-09-25 02:44:11 on dist-test-slave-n563"
server_key: "6b4a98535bae4dfe1392e15144a81316"
server_key_iv: "ceda318662e95691e69d972f1dcd00c0"
server_key_version: "encryptionkey@0"
I20260925 02:44:11.734259 9565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "5ba55afe232b45889dd3fc45f35d5f08"
format_stamp: "Formatted at 2026-09-25 02:44:11 on dist-test-slave-n563"
server_key: "6b4a98535bae4dfe1392e15144a81316"
server_key_iv: "ceda318662e95691e69d972f1dcd00c0"
server_key_version: "encryptionkey@0"
I20260925 02:44:11.740109 9565 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:44:11.743924 9579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:11.745411 9565 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:44:11.745616 9565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "5ba55afe232b45889dd3fc45f35d5f08"
format_stamp: "Formatted at 2026-09-25 02:44:11 on dist-test-slave-n563"
server_key: "6b4a98535bae4dfe1392e15144a81316"
server_key_iv: "ceda318662e95691e69d972f1dcd00c0"
server_key_version: "encryptionkey@0"
I20260925 02:44:11.745865 9565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:11.760715 9565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:11.765331 9565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:11.765645 9565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:11.766732 9565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:11.768168 9565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:11.768257 9565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:11.768399 9565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:11.768507 9565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:11.801528 9565 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:35775
I20260925 02:44:11.801579 9693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:35775 every 8 connection(s)
I20260925 02:44:11.802512 9565 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:44:11.807149 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 9565
I20260925 02:44:11.807366 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:44:11.808019 5346 external_mini_cluster.cc:1428] Setting key 4160b279718467d439b8cb7b6e82393c
Sep 25 02:44:11 dist-test-slave-n563 krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304251, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:11.825843 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:11.804987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:59555 (local address 127.5.56.190:46667)
0925 02:44:11.805474 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:11.805479 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:11.806299 (+ 820us) server_negotiation.cc:407] Connection header received
0925 02:44:11.808364 (+ 2065us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:11.808370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:11.808462 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:11.808604 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:11.810578 (+ 1974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:11.811305 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:11.813911 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:11.814322 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:11.814497 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:11.818637 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:11.818647 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:11.818655 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:11.818703 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:11.822292 (+ 3589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:11.823168 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:11.823176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:11.823183 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:11.823342 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:11.823765 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:11.823772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:11.823778 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:11.824185 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:11.824339 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:11.825051 (+ 712us) server_negotiation.cc:299] Negotiation successful
0925 02:44:11.825338 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":261,"threads_started":1}
I20260925 02:44:11.826577 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:11.805413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:46667 (local address 127.5.56.129:59555)
0925 02:44:11.806125 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:11.806171 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:11.808040 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:11.808871 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:11.808883 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:11.809488 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:11.810359 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:11.810415 (+ 56us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:11.811469 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:11.811477 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:11.813737 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:11.813756 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:11.814614 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:11.814624 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:11.814952 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:11.815736 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:11.815777 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:11.818311 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:11.822517 (+ 4206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:11.822524 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:11.822539 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:11.823017 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:11.823456 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:11.823460 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:11.823462 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:11.823643 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:11.824306 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:11.824312 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:11.824648 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 02:44:11.824881 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 02:44:11.825330 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":276,"threads_started":1}
I20260925 02:44:11.828205 9694 heartbeater.cc:347] Connected to a master server at 127.5.56.190:46667
I20260925 02:44:11.828564 9694 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:11.829492 9694 heartbeater.cc:511] Master 127.5.56.190:46667 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
I20260925 02:44:11.832013 9504 ts_manager.cc:194] Registered new tserver with Master: 5ba55afe232b45889dd3fc45f35d5f08 (127.5.56.129:35775)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
I20260925 02:44:11.833720 9504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:59555
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:44:11.858309 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:46667
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:12.193600 9700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:12.194044 9700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:12.194216 9700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:12.202263 9700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:12.202445 9700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:12.202647 9700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:44:12.212901 9700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:46667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:12.214361 9700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:12.215757 9700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:12.230243 9705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:12.230280 9706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:12.230167 9708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:12.231382 9700 server_base.cc:1034] running on GCE node
I20260925 02:44:12.232193 9700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:12.233270 9700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:12.234520 9700 hybrid_clock.cc:648] HybridClock initialized: now 1790304252234465 us; error 66 us; skew 500 ppm
I20260925 02:44:12.234911 9700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:12 dist-test-slave-n563 krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304252, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:12.238373 9700 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:44:12.239673 9700 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:12.239773 9700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:12.240077 9700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:12.243398 9700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "252c4f6cda9840f1b9b810707f175bf0"
format_stamp: "Formatted at 2026-09-25 02:44:12 on dist-test-slave-n563"
server_key: "6d1db79c1936c99815c88da83a3c2386"
server_key_iv: "52e440e0d46250ef43f66b808744c884"
server_key_version: "encryptionkey@0"
I20260925 02:44:12.244237 9700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "252c4f6cda9840f1b9b810707f175bf0"
format_stamp: "Formatted at 2026-09-25 02:44:12 on dist-test-slave-n563"
server_key: "6d1db79c1936c99815c88da83a3c2386"
server_key_iv: "52e440e0d46250ef43f66b808744c884"
server_key_version: "encryptionkey@0"
I20260925 02:44:12.249740 9700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:44:12.253621 9714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:12.255147 9700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:44:12.255348 9700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "252c4f6cda9840f1b9b810707f175bf0"
format_stamp: "Formatted at 2026-09-25 02:44:12 on dist-test-slave-n563"
server_key: "6d1db79c1936c99815c88da83a3c2386"
server_key_iv: "52e440e0d46250ef43f66b808744c884"
server_key_version: "encryptionkey@0"
I20260925 02:44:12.255594 9700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:12.275780 9700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:12.280326 9700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:12.280632 9700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:12.281687 9700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:12.283252 9700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:12.283397 9700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:12.283551 9700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:12.283663 9700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:12.321372 9700 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:33559
I20260925 02:44:12.321416 9827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:33559 every 8 connection(s)
I20260925 02:44:12.322361 9700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:44:12.328838 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 9700
I20260925 02:44:12.329033 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:44:12.329458 5346 external_mini_cluster.cc:1428] Setting key 47379db6331ce3b23fe2a782101609ac
Sep 25 02:44:12 dist-test-slave-n563 krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304252, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:12.344575 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.325018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:54229 (local address 127.5.56.190:46667)
0925 02:44:12.325220 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:12.325225 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:12.326487 (+ 1262us) server_negotiation.cc:407] Connection header received
0925 02:44:12.328325 (+ 1838us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:12.328330 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:12.328426 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:12.328551 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:12.330654 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.331341 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.333086 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.333891 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.334034 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.338209 (+ 4175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:12.338214 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:12.338218 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:12.338249 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:12.340944 (+ 2695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:12.341994 (+ 1050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:12.341998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:12.342001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:12.342132 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:12.342705 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:12.342708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:12.342710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:12.342974 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:12.343083 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:12.344137 (+ 1054us) server_negotiation.cc:299] Negotiation successful
0925 02:44:12.344312 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:44:12.346428 9830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.325554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:46667 (local address 127.5.56.130:54229)
0925 02:44:12.326299 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:12.326337 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:12.328052 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:12.328824 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:12.328833 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:12.329522 (+ 689us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:12.330448 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.330481 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.331506 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.331515 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.332924 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.332937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.334104 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.334109 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.334315 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.335131 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:12.335156 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:12.337962 (+ 2806us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:12.341094 (+ 3132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:12.341107 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:12.341128 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:12.341858 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:12.342261 (+ 403us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:12.342269 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:12.342276 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:12.342578 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:12.343116 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:12.343127 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:12.343713 (+ 586us) client_negotiation.cc:769] Sending connection context
0925 02:44:12.344050 (+ 337us) client_negotiation.cc:240] Negotiation successful
0925 02:44:12.344714 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":312,"threads_started":1}
I20260925 02:44:12.347994 9828 heartbeater.cc:347] Connected to a master server at 127.5.56.190:46667
I20260925 02:44:12.348382 9828 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:12.349321 9828 heartbeater.cc:511] Master 127.5.56.190:46667 requested a full tablet report, sending...
I20260925 02:44:12.351359 9504 ts_manager.cc:194] Registered new tserver with Master: 252c4f6cda9840f1b9b810707f175bf0 (127.5.56.130:33559)
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
I20260925 02:44:12.352259 9504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:54229
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:44:12.381383 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:46667
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:12.712944 9834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:12.713227 9834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:12.713322 9834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:12.720793 9834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:12.720907 9834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:12.721021 9834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:44:12.730893 9834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:46667
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:12.732231 9834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:12.733560 9834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:12.747159 9839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:12.747073 9840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:12.747380 9842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:12.748181 9834 server_base.cc:1034] running on GCE node
I20260925 02:44:12.748730 9834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:12.749747 9834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:12.750945 9834 hybrid_clock.cc:648] HybridClock initialized: now 1790304252750904 us; error 39 us; skew 500 ppm
I20260925 02:44:12.751238 9834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:12 dist-test-slave-n563 krb5kdc[9459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304252, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:12.754503 9834 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:44:12.755570 9834 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:12.755661 9834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:12.755926 9834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:12.759013 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "0dee024969bc45aaaf293bc3f53ec643"
format_stamp: "Formatted at 2026-09-25 02:44:12 on dist-test-slave-n563"
server_key: "6b02557583caed0db72f6de7e9a93256"
server_key_iv: "54c25f4234cae3bf29263c38d8d0bae4"
server_key_version: "encryptionkey@0"
I20260925 02:44:12.759904 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "0dee024969bc45aaaf293bc3f53ec643"
format_stamp: "Formatted at 2026-09-25 02:44:12 on dist-test-slave-n563"
server_key: "6b02557583caed0db72f6de7e9a93256"
server_key_iv: "54c25f4234cae3bf29263c38d8d0bae4"
server_key_version: "encryptionkey@0"
I20260925 02:44:12.765170 9834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:44:12.768788 9848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:12.770196 9834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:12.770452 9834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "0dee024969bc45aaaf293bc3f53ec643"
format_stamp: "Formatted at 2026-09-25 02:44:12 on dist-test-slave-n563"
server_key: "6b02557583caed0db72f6de7e9a93256"
server_key_iv: "54c25f4234cae3bf29263c38d8d0bae4"
server_key_version: "encryptionkey@0"
I20260925 02:44:12.770697 9834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:12.797435 9834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:12.801774 9834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:12.802060 9834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:12.803155 9834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:12.804642 9834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:12.804762 9834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:12.804891 9834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:12.804978 9834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:12.836966 9694 heartbeater.cc:503] Master 127.5.56.190:46667 was elected leader, sending a full tablet report...
I20260925 02:44:12.838462 9834 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:46001
I20260925 02:44:12.838505 9960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:46001 every 8 connection(s)
I20260925 02:44:12.839438 9834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:44:12.841984 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 9834
I20260925 02:44:12.842193 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:44:12.842734 5346 external_mini_cluster.cc:1428] Setting key 41287f5fa9e0c7279d0547cdc383187c
Sep 25 02:44:12 dist-test-slave-n563 krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304252, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:12.860095 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.841945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:59773 (local address 127.5.56.190:46667)
0925 02:44:12.842543 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:12.842548 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:12.843711 (+ 1163us) server_negotiation.cc:407] Connection header received
0925 02:44:12.846079 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:12.846085 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:12.846186 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:12.846300 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:12.847965 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.848684 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.850243 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.850738 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.850870 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.854157 (+ 3287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:12.854162 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:12.854165 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:12.854196 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:12.857153 (+ 2957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:12.857829 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:12.857837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:12.857842 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:12.858007 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:12.858418 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:12.858424 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:12.858427 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:12.858692 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:12.858851 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:12.859518 (+ 667us) server_negotiation.cc:299] Negotiation successful
0925 02:44:12.859750 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481}
I20260925 02:44:12.861156 9963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.842674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:46667 (local address 127.5.56.131:59773)
0925 02:44:12.843545 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:12.843593 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:12.845795 (+ 2202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:12.846576 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:12.846583 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:12.847248 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:12.847794 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.847811 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.848832 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.848838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.850097 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.850108 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.850976 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.850994 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.851211 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.851831 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:12.851857 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:12.853950 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:12.857271 (+ 3321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:12.857277 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:12.857290 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:12.857704 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:12.858119 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:12.858124 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:12.858129 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:12.858292 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:12.858825 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:12.858832 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:12.859164 (+ 332us) client_negotiation.cc:769] Sending connection context
0925 02:44:12.859436 (+ 272us) client_negotiation.cc:240] Negotiation successful
0925 02:44:12.859955 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":337,"threads_started":1}
I20260925 02:44:12.862524 9961 heartbeater.cc:347] Connected to a master server at 127.5.56.190:46667
I20260925 02:44:12.862852 9961 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:12.863517 9961 heartbeater.cc:511] Master 127.5.56.190:46667 requested a full tablet report, sending...
I20260925 02:44:12.864946 9504 ts_manager.cc:194] Registered new tserver with Master: 0dee024969bc45aaaf293bc3f53ec643 (127.5.56.131:46001)
I20260925 02:44:12.865808 9504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:59773
I20260925 02:44:12.870332 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:44:12.886904 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.874912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.5.56.190:46667)
0925 02:44:12.875101 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:12.875106 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:12.875676 (+ 570us) server_negotiation.cc:407] Connection header received
0925 02:44:12.875811 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:12.875814 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:12.875875 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:12.875954 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:12.877279 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.877962 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.879286 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.879836 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.879960 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.882181 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:12.882186 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:12.882189 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:12.882220 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:12.884639 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:12.885243 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:12.885247 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:12.885249 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:12.885366 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:12.885668 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:12.885671 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:12.885673 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:12.885967 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:12.886148 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:12.886519 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 02:44:12.886676 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:44:12.891116 9504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36200:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:44:12.898284 9504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:44:12.922729 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.911182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.5.56.131:46001)
0925 02:44:12.913099 (+ 1917us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:12.913111 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:12.913896 (+ 785us) server_negotiation.cc:407] Connection header received
0925 02:44:12.914020 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:12.914031 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:12.914316 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:12.915449 (+ 1133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:12.915770 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.917456 (+ 1686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.920374 (+ 2918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.921516 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.921674 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.921916 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:12.922172 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 02:44:12.922336 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1682,"thread_start_us":1473,"threads_started":1}
I20260925 02:44:12.922806 9976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.910689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:46001 (local address 127.0.0.1:38700)
0925 02:44:12.913603 (+ 2914us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:12.913627 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:12.913762 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:12.914695 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:12.914706 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:12.914750 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:12.915250 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.915266 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.917601 (+ 2335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.917606 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.920196 (+ 2590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.920211 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.921704 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.921710 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.921904 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.921961 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 02:44:12.922117 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:44:12.922315 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2765,"mutex_wait_us":219,"thread_start_us":2443,"threads_started":1}
I20260925 02:44:12.923370 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.913241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:35775 (local address 127.0.0.1:43804)
0925 02:44:12.913836 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:12.913853 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:12.913946 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:12.916479 (+ 2533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:12.916484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:12.916506 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:12.916927 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.916940 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.918842 (+ 1902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.918852 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.920734 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.920751 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.922513 (+ 1762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.922523 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.922726 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.922764 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 02:44:12.922830 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 02:44:12.922974 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"mutex_wait_us":1,"thread_start_us":776,"threads_started":1}
I20260925 02:44:12.924348 9974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.910685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:33559 (local address 127.0.0.1:47246)
0925 02:44:12.911417 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:12.911458 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:12.911620 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:12.916191 (+ 4571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:12.916194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:12.916215 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:12.916636 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.916653 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.918252 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.918263 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.920556 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.920573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.923571 (+ 2998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.923578 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.923728 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.923756 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:44:12.923831 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:44:12.923942 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":88,"thread_start_us":348,"threads_started":1}
I20260925 02:44:12.924476 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.914002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.5.56.129:35775)
0925 02:44:12.914714 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:12.914722 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:12.914748 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:44:12.914845 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:12.914853 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:12.915084 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:12.916471 (+ 1387us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:12.917126 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.918668 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.920898 (+ 2230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.922243 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.923774 (+ 1531us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.923951 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:12.924005 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 02:44:12.924139 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":356,"threads_started":1}
I20260925 02:44:12.925102 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.910885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47246 (local address 127.5.56.130:33559)
0925 02:44:12.915629 (+ 4744us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:12.915639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:12.915666 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:44:12.915769 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:12.915776 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:12.916000 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:12.916198 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:12.916807 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.918065 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.920720 (+ 2655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.922197 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.924410 (+ 2213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.924637 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:12.924718 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 02:44:12.924871 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4569,"thread_start_us":4463,"threads_started":1}
I20260925 02:44:12.927412 9896 tablet_service.cc:1467] Processing CreateTablet for tablet c59068512087453c930ddf132f21f163 (DEFAULT_TABLE table=test-table [id=39a7bad0335a49c5a6148235242e7ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:12.927639 9627 tablet_service.cc:1467] Processing CreateTablet for tablet c59068512087453c930ddf132f21f163 (DEFAULT_TABLE table=test-table [id=39a7bad0335a49c5a6148235242e7ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:12.928495 9763 tablet_service.cc:1467] Processing CreateTablet for tablet c59068512087453c930ddf132f21f163 (DEFAULT_TABLE table=test-table [id=39a7bad0335a49c5a6148235242e7ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:12.929222 9627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59068512087453c930ddf132f21f163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:12.929342 9896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59068512087453c930ddf132f21f163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:12.929788 9763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c59068512087453c930ddf132f21f163. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:12.940583 9980 tablet_bootstrap.cc:492] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08: Bootstrap starting.
I20260925 02:44:12.944641 9980 tablet_bootstrap.cc:654] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:12.944700 9981 tablet_bootstrap.cc:492] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0: Bootstrap starting.
I20260925 02:44:12.946187 9982 tablet_bootstrap.cc:492] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643: Bootstrap starting.
I20260925 02:44:12.946271 9980 log.cc:824] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:12.948432 9981 tablet_bootstrap.cc:654] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:12.949290 9980 tablet_bootstrap.cc:492] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08: No bootstrap required, opened a new log
I20260925 02:44:12.949620 9980 ts_tablet_manager.cc:1397] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260925 02:44:12.949687 9981 log.cc:824] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:12.949992 9982 tablet_bootstrap.cc:654] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:12.951501 9982 log.cc:824] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:12.955322 9982 tablet_bootstrap.cc:492] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643: No bootstrap required, opened a new log
I20260925 02:44:12.955677 9982 ts_tablet_manager.cc:1397] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260925 02:44:12.957921 9980 raft_consensus.cc:348] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252c4f6cda9840f1b9b810707f175bf0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 33559 } } peers { permanent_uuid: "0dee024969bc45aaaf293bc3f53ec643" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46001 } } peers { permanent_uuid: "5ba55afe232b45889dd3fc45f35d5f08" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35775 } }
I20260925 02:44:12.958158 9981 tablet_bootstrap.cc:492] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0: No bootstrap required, opened a new log
I20260925 02:44:12.958279 9980 raft_consensus.cc:374] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:12.958439 9981 ts_tablet_manager.cc:1397] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260925 02:44:12.958421 9980 raft_consensus.cc:729] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ba55afe232b45889dd3fc45f35d5f08, State: Initialized, Role: FOLLOWER
I20260925 02:44:12.959043 9980 consensus_queue.cc:260] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252c4f6cda9840f1b9b810707f175bf0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 33559 } } peers { permanent_uuid: "0dee024969bc45aaaf293bc3f53ec643" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46001 } } peers { permanent_uuid: "5ba55afe232b45889dd3fc45f35d5f08" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35775 } }
I20260925 02:44:12.960750 9980 ts_tablet_manager.cc:1428] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260925 02:44:12.965504 9982 raft_consensus.cc:348] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252c4f6cda9840f1b9b810707f175bf0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 33559 } } peers { permanent_uuid: "0dee024969bc45aaaf293bc3f53ec643" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46001 } } peers { permanent_uuid: "5ba55afe232b45889dd3fc45f35d5f08" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35775 } }
I20260925 02:44:12.965894 9982 raft_consensus.cc:374] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:12.966017 9982 raft_consensus.cc:729] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dee024969bc45aaaf293bc3f53ec643, State: Initialized, Role: FOLLOWER
I20260925 02:44:12.966791 9982 consensus_queue.cc:260] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252c4f6cda9840f1b9b810707f175bf0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 33559 } } peers { permanent_uuid: "0dee024969bc45aaaf293bc3f53ec643" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46001 } } peers { permanent_uuid: "5ba55afe232b45889dd3fc45f35d5f08" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35775 } }
I20260925 02:44:12.967914 9961 heartbeater.cc:503] Master 127.5.56.190:46667 was elected leader, sending a full tablet report...
I20260925 02:44:12.968513 9982 ts_tablet_manager.cc:1428] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643: Time spent starting tablet: real 0.013s user 0.005s sys 0.008s
I20260925 02:44:12.968905 9981 raft_consensus.cc:348] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252c4f6cda9840f1b9b810707f175bf0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 33559 } } peers { permanent_uuid: "0dee024969bc45aaaf293bc3f53ec643" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46001 } } peers { permanent_uuid: "5ba55afe232b45889dd3fc45f35d5f08" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35775 } }
I20260925 02:44:12.969385 9981 raft_consensus.cc:374] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:12.969506 9981 raft_consensus.cc:729] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 252c4f6cda9840f1b9b810707f175bf0, State: Initialized, Role: FOLLOWER
I20260925 02:44:12.970132 9981 consensus_queue.cc:260] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252c4f6cda9840f1b9b810707f175bf0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 33559 } } peers { permanent_uuid: "0dee024969bc45aaaf293bc3f53ec643" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46001 } } peers { permanent_uuid: "5ba55afe232b45889dd3fc45f35d5f08" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35775 } }
I20260925 02:44:12.971268 9828 heartbeater.cc:503] Master 127.5.56.190:46667 was elected leader, sending a full tablet report...
I20260925 02:44:12.971843 9981 ts_tablet_manager.cc:1428] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0: Time spent starting tablet: real 0.013s user 0.010s sys 0.001s
I20260925 02:44:12.973896 9988 raft_consensus.cc:482] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:44:12.974253 9988 raft_consensus.cc:504] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252c4f6cda9840f1b9b810707f175bf0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 33559 } } peers { permanent_uuid: "0dee024969bc45aaaf293bc3f53ec643" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46001 } } peers { permanent_uuid: "5ba55afe232b45889dd3fc45f35d5f08" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35775 } }
I20260925 02:44:12.976501 9988 leader_election.cc:288] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0dee024969bc45aaaf293bc3f53ec643 (127.5.56.131:46001), 5ba55afe232b45889dd3fc45f35d5f08 (127.5.56.129:35775)
I20260925 02:44:12.983084 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.976957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:40993 (local address 127.5.56.129:35775)
0925 02:44:12.977209 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:12.977220 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:12.977787 (+ 567us) server_negotiation.cc:407] Connection header received
0925 02:44:12.978019 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:12.978022 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:12.978082 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:12.978190 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:12.978822 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.979590 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.981294 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.982100 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.982306 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.982382 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:12.982698 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 02:44:12.982822 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:44:12.983178 9989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.976937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:35775 (local address 127.5.56.130:40993)
0925 02:44:12.977679 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:12.977698 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:12.977845 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:12.978214 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:12.978219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:12.978255 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:12.978682 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.978692 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.979715 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.979723 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.981150 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.981161 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.982319 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.982325 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.982542 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.982566 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:44:12.982673 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:44:12.982798 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":408,"threads_started":1}
I20260925 02:44:12.983911 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.977554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:52927 (local address 127.5.56.131:46001)
0925 02:44:12.977708 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:12.977711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:12.978607 (+ 896us) server_negotiation.cc:407] Connection header received
0925 02:44:12.978794 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:12.978799 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:12.978867 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:12.978959 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:12.979473 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.980178 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.981806 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.982800 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.982985 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.983057 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:12.983538 (+ 481us) server_negotiation.cc:299] Negotiation successful
0925 02:44:12.983672 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 02:44:12.983938 9990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:12.976947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:46001 (local address 127.5.56.130:52927)
0925 02:44:12.978499 (+ 1552us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:12.978518 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:12.978664 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:12.978973 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:12.978977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:12.978999 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:12.979333 (+ 334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.979341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.980314 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.980319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.981660 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:12.981670 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:12.983017 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:12.983023 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:12.983349 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:12.983380 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:44:12.983511 (+ 131us) client_negotiation.cc:240] Negotiation successful
0925 02:44:12.983632 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1403,"thread_start_us":1195,"threads_started":1}
I20260925 02:44:12.984185 9647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59068512087453c930ddf132f21f163" candidate_uuid: "252c4f6cda9840f1b9b810707f175bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba55afe232b45889dd3fc45f35d5f08" is_pre_election: true
I20260925 02:44:12.984791 9647 raft_consensus.cc:2454] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252c4f6cda9840f1b9b810707f175bf0 in term 0.
I20260925 02:44:12.985028 9916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59068512087453c930ddf132f21f163" candidate_uuid: "252c4f6cda9840f1b9b810707f175bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dee024969bc45aaaf293bc3f53ec643" is_pre_election: true
I20260925 02:44:12.985533 9916 raft_consensus.cc:2454] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 252c4f6cda9840f1b9b810707f175bf0 in term 0.
I20260925 02:44:12.985499 9715 leader_election.cc:302] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 252c4f6cda9840f1b9b810707f175bf0, 5ba55afe232b45889dd3fc45f35d5f08; no voters:
I20260925 02:44:12.985931 9988 raft_consensus.cc:2781] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:44:12.986114 9988 raft_consensus.cc:482] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:44:12.986254 9988 raft_consensus.cc:3037] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:12.988665 9988 raft_consensus.cc:504] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252c4f6cda9840f1b9b810707f175bf0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 33559 } } peers { permanent_uuid: "0dee024969bc45aaaf293bc3f53ec643" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46001 } } peers { permanent_uuid: "5ba55afe232b45889dd3fc45f35d5f08" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35775 } }
I20260925 02:44:12.989305 9988 leader_election.cc:288] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [CANDIDATE]: Term 1 election: Requested vote from peers 0dee024969bc45aaaf293bc3f53ec643 (127.5.56.131:46001), 5ba55afe232b45889dd3fc45f35d5f08 (127.5.56.129:35775)
I20260925 02:44:12.989773 9916 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59068512087453c930ddf132f21f163" candidate_uuid: "252c4f6cda9840f1b9b810707f175bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0dee024969bc45aaaf293bc3f53ec643"
I20260925 02:44:12.989856 9647 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c59068512087453c930ddf132f21f163" candidate_uuid: "252c4f6cda9840f1b9b810707f175bf0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ba55afe232b45889dd3fc45f35d5f08"
I20260925 02:44:12.989972 9916 raft_consensus.cc:3037] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:12.990113 9647 raft_consensus.cc:3037] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:12.992626 9916 raft_consensus.cc:2454] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252c4f6cda9840f1b9b810707f175bf0 in term 1.
I20260925 02:44:12.992652 9647 raft_consensus.cc:2454] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 252c4f6cda9840f1b9b810707f175bf0 in term 1.
I20260925 02:44:12.993044 9716 leader_election.cc:302] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0dee024969bc45aaaf293bc3f53ec643, 252c4f6cda9840f1b9b810707f175bf0; no voters:
I20260925 02:44:12.993395 9988 raft_consensus.cc:2781] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:12.993819 9988 raft_consensus.cc:686] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 1 LEADER]: Becoming Leader. State: Replica: 252c4f6cda9840f1b9b810707f175bf0, State: Running, Role: LEADER
I20260925 02:44:12.994349 9988 consensus_queue.cc:237] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252c4f6cda9840f1b9b810707f175bf0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 33559 } } peers { permanent_uuid: "0dee024969bc45aaaf293bc3f53ec643" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46001 } } peers { permanent_uuid: "5ba55afe232b45889dd3fc45f35d5f08" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35775 } }
I20260925 02:44:12.999013 9504 catalog_manager.cc:5495] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 252c4f6cda9840f1b9b810707f175bf0 (127.5.56.130). New cstate: current_term: 1 leader_uuid: "252c4f6cda9840f1b9b810707f175bf0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "252c4f6cda9840f1b9b810707f175bf0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 33559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0dee024969bc45aaaf293bc3f53ec643" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46001 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ba55afe232b45889dd3fc45f35d5f08" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35775 } health_report { overall_health: UNKNOWN } } }
I20260925 02:44:13.022557 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:13.017314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47250 (local address 127.5.56.130:33559)
0925 02:44:13.017559 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:13.017565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:13.017629 (+ 64us) server_negotiation.cc:407] Connection header received
0925 02:44:13.017807 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:13.017811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:13.017891 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:13.018016 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:44:13.018358 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:13.019116 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:13.020366 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:13.020755 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:13.020843 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:13.021264 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:44:13.021793 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:44:13.021902 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:13.022061 (+ 159us) server_negotiation.cc:299] Negotiation successful
0925 02:44:13.022193 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:44:13.031078 9647 raft_consensus.cc:1261] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08 [term 1 FOLLOWER]: Refusing update from remote peer 252c4f6cda9840f1b9b810707f175bf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:13.031087 9916 raft_consensus.cc:1261] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643 [term 1 FOLLOWER]: Refusing update from remote peer 252c4f6cda9840f1b9b810707f175bf0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:13.032083 9991 consensus_queue.cc:1035] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ba55afe232b45889dd3fc45f35d5f08" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:13.032447 9988 consensus_queue.cc:1035] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0dee024969bc45aaaf293bc3f53ec643" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:13.041513 9994 mvcc.cc:205] Tried to move back new op lower bound from 7333086220404191232 to 7333086220269625344. Current Snapshot: MvccSnapshot[applied={T|T < 7333086220404191232}]
I20260925 02:44:13.043772 9995 mvcc.cc:205] Tried to move back new op lower bound from 7333086220404191232 to 7333086220269625344. Current Snapshot: MvccSnapshot[applied={T|T < 7333086220404191232}]
I20260925 02:44:13.060762 9503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:13.061097 9503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36200:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:13.064953 9503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9163b4667ce34676b134d590abd6fb5b: Sending DeleteTablet for 3 replicas of tablet c59068512087453c930ddf132f21f163
I20260925 02:44:13.066260 9896 tablet_service.cc:1514] Processing DeleteTablet for tablet c59068512087453c930ddf132f21f163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:13 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:38700
I20260925 02:44:13.066350 9763 tablet_service.cc:1514] Processing DeleteTablet for tablet c59068512087453c930ddf132f21f163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:13 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:47246
I20260925 02:44:13.066344 9627 tablet_service.cc:1514] Processing DeleteTablet for tablet c59068512087453c930ddf132f21f163 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:13 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:43804
I20260925 02:44:13.068547 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 9565
I20260925 02:44:13.079686 10005 tablet_replica.cc:330] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08: stopping tablet replica
I20260925 02:44:13.080513 10005 raft_consensus.cc:2229] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:13.081291 10005 raft_consensus.cc:2258] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:13.084335 10005 ts_tablet_manager.cc:1905] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:13.087036 10004 tablet_replica.cc:330] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643: stopping tablet replica
I20260925 02:44:13.087704 10006 tablet_replica.cc:330] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0: stopping tablet replica
I20260925 02:44:13.088228 10006 raft_consensus.cc:2229] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:44:13.088745 10004 raft_consensus.cc:2229] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:13.089470 10006 raft_consensus.cc:2258] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:13.090158 10004 raft_consensus.cc:2258] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:13.093657 10005 ts_tablet_manager.cc:1918] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:13.093914 10005 log.cc:1196] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/c59068512087453c930ddf132f21f163
I20260925 02:44:13.093976 10004 ts_tablet_manager.cc:1905] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:13.094661 10005 ts_tablet_manager.cc:1939] T c59068512087453c930ddf132f21f163 P 5ba55afe232b45889dd3fc45f35d5f08: Deleting consensus metadata
Sep 25 02:44:13 dist-test-slave-n563 krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:44:13.095391 10006 ts_tablet_manager.cc:1905] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:13.096959 9488 catalog_manager.cc:4841] TS 5ba55afe232b45889dd3fc45f35d5f08 (127.5.56.129:35775): tablet c59068512087453c930ddf132f21f163 (table test-table [id=39a7bad0335a49c5a6148235242e7ced]) successfully deleted
I20260925 02:44:13.103480 10004 ts_tablet_manager.cc:1918] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:13.103777 10004 log.cc:1196] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/c59068512087453c930ddf132f21f163
I20260925 02:44:13.104367 10004 ts_tablet_manager.cc:1939] T c59068512087453c930ddf132f21f163 P 0dee024969bc45aaaf293bc3f53ec643: Deleting consensus metadata
I20260925 02:44:13.104445 9979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:13.069996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43814 (local address 127.5.56.129:35775)
0925 02:44:13.070191 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:13.070197 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:13.076602 (+ 6405us) server_negotiation.cc:407] Connection header received
0925 02:44:13.076837 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:13.076843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:13.076959 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:13.077069 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:13.086584 (+ 9515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:13.087738 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:13.089827 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:13.090604 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:13.090761 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:13.096608 (+ 5847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:13.096625 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:13.096643 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:13.096695 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:13.100583 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:13.101593 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:13.101605 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:13.101608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:13.101775 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:13.102293 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:13.102298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:13.102302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:13.102817 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:13.103149 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:13.103765 (+ 616us) server_negotiation.cc:299] Negotiation successful
0925 02:44:13.104035 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:44:13.104900 10006 ts_tablet_manager.cc:1918] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:13.105129 10006 log.cc:1196] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/c59068512087453c930ddf132f21f163
I20260925 02:44:13.105767 10006 ts_tablet_manager.cc:1939] T c59068512087453c930ddf132f21f163 P 252c4f6cda9840f1b9b810707f175bf0: Deleting consensus metadata
I20260925 02:44:13.106263 9689 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:13.106467 9489 catalog_manager.cc:4841] TS 0dee024969bc45aaaf293bc3f53ec643 (127.5.56.131:46001): tablet c59068512087453c930ddf132f21f163 (table test-table [id=39a7bad0335a49c5a6148235242e7ced]) successfully deleted
I20260925 02:44:13.107530 9490 catalog_manager.cc:4841] TS 252c4f6cda9840f1b9b810707f175bf0 (127.5.56.130:33559): tablet c59068512087453c930ddf132f21f163 (table test-table [id=39a7bad0335a49c5a6148235242e7ced]) successfully deleted
I20260925 02:44:13.198452 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 9565
I20260925 02:44:13.222684 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 9700
Sep 25 02:44:13 dist-test-slave-n563 krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:44:13.242285 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:13.223515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47256 (local address 127.5.56.130:33559)
0925 02:44:13.223711 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:13.223716 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:13.223773 (+ 57us) server_negotiation.cc:407] Connection header received
0925 02:44:13.223964 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:13.223968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:13.224043 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:13.224133 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:13.225740 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:13.227316 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:13.229608 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:13.230466 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:13.230600 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:13.235988 (+ 5388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:13.236004 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:13.236022 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:13.236080 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:13.239259 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:13.239990 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:13.240006 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:13.240008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:13.240174 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:13.240588 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:13.240599 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:13.240601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:13.240965 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:13.241327 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:13.241635 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 02:44:13.241872 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:44:13.243099 9823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:13.360239 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 9700
I20260925 02:44:13.380421 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 9834
Sep 25 02:44:13 dist-test-slave-n563 krb5kdc[9459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:44:13.395776 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:13.381257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.5.56.131:46001)
0925 02:44:13.381436 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:13.381440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:13.381460 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:44:13.381593 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:13.381597 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:13.381661 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:13.381744 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:13.383122 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:13.383831 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:13.385028 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:13.385404 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:13.385498 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:13.389907 (+ 4409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:13.389923 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:13.389940 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:13.389991 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:13.392788 (+ 2797us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:13.393484 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:13.393494 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:13.393496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:13.393690 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:13.394081 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:13.394086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:13.394089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:13.394559 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:13.394880 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:13.395207 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 02:44:13.395437 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:44:13.396457 9956 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:13.487484 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 9834
I20260925 02:44:13.507292 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 9473
I20260925 02:44:13.508529 9534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:13.618717 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 9473
2026-09-25T02:44:13Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4232 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T02:44:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:13Z Disabled control of system clock
I20260925 02:44:13.688027 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43065
--webserver_interface=127.5.56.190
--webserver_port=0
--builtin_ntp_servers=127.5.56.148:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:43065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:44:14.028167 10015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:14.028467 10015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:14.028625 10015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:14.037410 10015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:14.037559 10015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:14.037627 10015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:14.037717 10015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:14.037783 10015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:14.052778 10015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:43065
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:14.054847 10015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:14.057826 10015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:14.075794 10020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:14.075757 10021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:14.080721 10023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:14.080734 10015 server_base.cc:1034] running on GCE node
I20260925 02:44:14.081707 10015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:14.083501 10015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:14.084757 10015 hybrid_clock.cc:648] HybridClock initialized: now 1790304254084686 us; error 68 us; skew 500 ppm
I20260925 02:44:14.085101 10015 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:44:14.087519 10015 webserver.cc:460] Webserver started at http://127.5.56.190:36197/ using document root <none> and password file <none>
I20260925 02:44:14.088290 10015 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:14.088380 10015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:14.088680 10015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:14.091923 10015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "3c9b1d634208484ab077b7eafa4feaf4"
format_stamp: "Formatted at 2026-09-25 02:44:14 on dist-test-slave-n563"
server_key: "7186b5e150e8924358777a1fc84ed48a"
server_key_iv: "fc52c17adf5cfb7caca9d04e493de663"
server_key_version: "encryptionkey@0"
I20260925 02:44:14.092729 10015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "3c9b1d634208484ab077b7eafa4feaf4"
format_stamp: "Formatted at 2026-09-25 02:44:14 on dist-test-slave-n563"
server_key: "7186b5e150e8924358777a1fc84ed48a"
server_key_iv: "fc52c17adf5cfb7caca9d04e493de663"
server_key_version: "encryptionkey@0"
I20260925 02:44:14.098307 10015 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 02:44:14.102154 10029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:14.103801 10015 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 02:44:14.104023 10015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "3c9b1d634208484ab077b7eafa4feaf4"
format_stamp: "Formatted at 2026-09-25 02:44:14 on dist-test-slave-n563"
server_key: "7186b5e150e8924358777a1fc84ed48a"
server_key_iv: "fc52c17adf5cfb7caca9d04e493de663"
server_key_version: "encryptionkey@0"
I20260925 02:44:14.104291 10015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:14.110486 10015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:14.111316 10015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:14.111634 10015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:14.129531 10015 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:43065
I20260925 02:44:14.129568 10080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:43065 every 8 connection(s)
I20260925 02:44:14.131036 10015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:14.135345 10081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:14.139456 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 10015
I20260925 02:44:14.139681 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:14.140321 5346 external_mini_cluster.cc:1428] Setting key 5bac9fcb7ac2b869725d5035e264fea0
I20260925 02:44:14.147034 10081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4: Bootstrap starting.
I20260925 02:44:14.149304 10084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:14.142502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53668 (local address 127.5.56.190:43065)
0925 02:44:14.143599 (+ 1097us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:14.143611 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:14.143654 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:44:14.145484 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:14.145511 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:14.145963 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:14.146372 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:14.147527 (+ 1155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:14.147534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:14.147546 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:14.147560 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:14.147626 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:14.147642 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:14.147887 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:14.148035 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 02:44:14.148305 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642,"thread_start_us":378,"threads_started":1}
I20260925 02:44:14.150813 10081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:14.151916 10081 log.cc:824] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:14.155139 10081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4: No bootstrap required, opened a new log
I20260925 02:44:14.162573 10081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9b1d634208484ab077b7eafa4feaf4" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43065 } }
I20260925 02:44:14.162982 10081 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:14.163101 10081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c9b1d634208484ab077b7eafa4feaf4, State: Initialized, Role: FOLLOWER
I20260925 02:44:14.163748 10081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9b1d634208484ab077b7eafa4feaf4" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43065 } }
I20260925 02:44:14.164000 10081 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:14.164162 10081 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:14.164347 10081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:14.166666 10081 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9b1d634208484ab077b7eafa4feaf4" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43065 } }
I20260925 02:44:14.167304 10081 leader_election.cc:302] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c9b1d634208484ab077b7eafa4feaf4; no voters:
I20260925 02:44:14.168077 10081 leader_election.cc:288] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:14.168190 10086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:14.168912 10086 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [term 1 LEADER]: Becoming Leader. State: Replica: 3c9b1d634208484ab077b7eafa4feaf4, State: Running, Role: LEADER
I20260925 02:44:14.169489 10086 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9b1d634208484ab077b7eafa4feaf4" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43065 } }
I20260925 02:44:14.170559 10081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:14.172683 10087 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c9b1d634208484ab077b7eafa4feaf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9b1d634208484ab077b7eafa4feaf4" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43065 } } }
I20260925 02:44:14.173069 10087 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:14.172739 10088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c9b1d634208484ab077b7eafa4feaf4. Latest consensus state: current_term: 1 leader_uuid: "3c9b1d634208484ab077b7eafa4feaf4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c9b1d634208484ab077b7eafa4feaf4" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43065 } } }
I20260925 02:44:14.173204 10088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:14.180136 10092 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:14.185708 10092 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:14.194656 10092 catalog_manager.cc:1347] Generated new cluster ID: 56e7a7d8a9924759bb10c0e0f1bba8f5
I20260925 02:44:14.194798 10092 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:14.228410 10092 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:14.228981 10092 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:14.239037 10092 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4: Generated new TSK 0
I20260925 02:44:14.240186 10092 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:44:14.250481 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43065
--builtin_ntp_servers=127.5.56.148:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 02:44:14.599076 10105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:14.599465 10105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:14.599647 10105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:14.607481 10105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:14.607607 10105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:14.607782 10105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:44:14.617889 10105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:14.619405 10105 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:14.621343 10105 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:14.635829 10110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:14.635937 10111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:14.635883 10113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:14.636605 10105 server_base.cc:1034] running on GCE node
I20260925 02:44:14.637317 10105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:14.638340 10105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:14.639611 10105 hybrid_clock.cc:648] HybridClock initialized: now 1790304254639548 us; error 69 us; skew 500 ppm
I20260925 02:44:14.639963 10105 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:44:14.648345 10105 webserver.cc:460] Webserver started at http://127.5.56.129:37549/ using document root <none> and password file <none>
I20260925 02:44:14.649190 10105 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:14.649344 10105 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:14.649751 10105 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:14.653064 10105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "35d1b7db8df545ab8022be46fde2b986"
format_stamp: "Formatted at 2026-09-25 02:44:14 on dist-test-slave-n563"
server_key: "3553a7739708810e0185c80efa1e85cd"
server_key_iv: "8d154608a01e128c1c2c9ba55a17e7e1"
server_key_version: "encryptionkey@0"
I20260925 02:44:14.653927 10105 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "35d1b7db8df545ab8022be46fde2b986"
format_stamp: "Formatted at 2026-09-25 02:44:14 on dist-test-slave-n563"
server_key: "3553a7739708810e0185c80efa1e85cd"
server_key_iv: "8d154608a01e128c1c2c9ba55a17e7e1"
server_key_version: "encryptionkey@0"
I20260925 02:44:14.659566 10105 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 02:44:14.663383 10120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:14.664835 10105 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:44:14.665019 10105 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "35d1b7db8df545ab8022be46fde2b986"
format_stamp: "Formatted at 2026-09-25 02:44:14 on dist-test-slave-n563"
server_key: "3553a7739708810e0185c80efa1e85cd"
server_key_iv: "8d154608a01e128c1c2c9ba55a17e7e1"
server_key_version: "encryptionkey@0"
I20260925 02:44:14.665287 10105 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:14.671300 10105 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:14.672154 10105 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:14.672446 10105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:14.673382 10105 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:14.674918 10105 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:14.675052 10105 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:14.675191 10105 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:14.675280 10105 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:14.709350 10105 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:34471
I20260925 02:44:14.709407 10232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:34471 every 8 connection(s)
I20260925 02:44:14.710870 10105 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:44:14.712981 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 10105
I20260925 02:44:14.713173 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:44:14.714012 5346 external_mini_cluster.cc:1428] Setting key 1f798d59bd22ab242bafe224d034afe7
I20260925 02:44:14.717188 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43065
--builtin_ntp_servers=127.5.56.148:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:44:14.720666 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:14.713999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:52065 (local address 127.5.56.190:43065)
0925 02:44:14.714780 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:14.714786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:14.716175 (+ 1389us) server_negotiation.cc:407] Connection header received
0925 02:44:14.718109 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:14.718118 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:14.718195 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:14.718321 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:14.719743 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:14.719752 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:14.719760 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:14.719785 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:14.719847 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:14.719867 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:14.719998 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:14.720186 (+ 188us) server_negotiation.cc:299] Negotiation successful
0925 02:44:14.720261 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":537,"threads_started":1}
I20260925 02:44:14.721617 10236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:14.714775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43065 (local address 127.5.56.129:52065)
0925 02:44:14.715969 (+ 1194us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:14.716039 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:14.717797 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:14.718593 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:14.718601 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:14.718705 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:14.719415 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:44:14.719417 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:14.719458 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:44:14.719553 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:14.719964 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:14.719971 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:14.719999 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:44:14.720220 (+ 221us) client_negotiation.cc:240] Negotiation successful
0925 02:44:14.720440 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":416,"threads_started":1}
I20260925 02:44:14.723326 10233 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43065
I20260925 02:44:14.723666 10233 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:14.724318 10233 heartbeater.cc:511] Master 127.5.56.190:43065 requested a full tablet report, sending...
I20260925 02:44:14.726543 10046 ts_manager.cc:194] Registered new tserver with Master: 35d1b7db8df545ab8022be46fde2b986 (127.5.56.129:34471)
W20260925 02:44:15.056131 10237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:15.056489 10237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:15.056663 10237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:15.064064 10237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:15.064245 10237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:15.064445 10237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:44:15.074467 10237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:15.075830 10237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:15.077674 10237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:15.092082 10242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:15.092186 10243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:15.092121 10245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:15.093233 10237 server_base.cc:1034] running on GCE node
I20260925 02:44:15.093961 10237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:15.095108 10237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:15.096318 10237 hybrid_clock.cc:648] HybridClock initialized: now 1790304255096268 us; error 42 us; skew 500 ppm
I20260925 02:44:15.096664 10237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:44:15.099172 10237 webserver.cc:460] Webserver started at http://127.5.56.130:36503/ using document root <none> and password file <none>
I20260925 02:44:15.099920 10237 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:15.100021 10237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:15.100340 10237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:15.103433 10237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "c9c72144132046c0be7869bf84719541"
format_stamp: "Formatted at 2026-09-25 02:44:15 on dist-test-slave-n563"
server_key: "f2e017a5f6eefc207af1c17cb419f1cc"
server_key_iv: "9af2de2d9d02a1ac3bdbe54d9b7dab7c"
server_key_version: "encryptionkey@0"
I20260925 02:44:15.104291 10237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "c9c72144132046c0be7869bf84719541"
format_stamp: "Formatted at 2026-09-25 02:44:15 on dist-test-slave-n563"
server_key: "f2e017a5f6eefc207af1c17cb419f1cc"
server_key_iv: "9af2de2d9d02a1ac3bdbe54d9b7dab7c"
server_key_version: "encryptionkey@0"
I20260925 02:44:15.110016 10237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:44:15.113816 10251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:15.115345 10237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:44:15.115532 10237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "c9c72144132046c0be7869bf84719541"
format_stamp: "Formatted at 2026-09-25 02:44:15 on dist-test-slave-n563"
server_key: "f2e017a5f6eefc207af1c17cb419f1cc"
server_key_iv: "9af2de2d9d02a1ac3bdbe54d9b7dab7c"
server_key_version: "encryptionkey@0"
I20260925 02:44:15.115816 10237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:15.121688 10237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:15.122602 10237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:15.122834 10237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:15.123737 10237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:15.125156 10237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:15.125248 10237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:15.125324 10237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:15.125378 10237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:15.162865 10237 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:41187
I20260925 02:44:15.162914 10364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:41187 every 8 connection(s)
I20260925 02:44:15.164237 10237 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:44:15.169428 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 10237
I20260925 02:44:15.169610 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:44:15.170161 5346 external_mini_cluster.cc:1428] Setting key d8ca3d8fdcc4d60a50dbeb569e33dbe6
I20260925 02:44:15.173659 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.166955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:33303 (local address 127.5.56.190:43065)
0925 02:44:15.167132 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:15.167137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:15.168651 (+ 1514us) server_negotiation.cc:407] Connection header received
0925 02:44:15.171145 (+ 2494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:15.171151 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:15.171255 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:15.171378 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:15.172987 (+ 1609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:15.172998 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:15.173001 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:15.173017 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:15.173051 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:15.173064 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:15.173156 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:15.173394 (+ 238us) server_negotiation.cc:299] Negotiation successful
0925 02:44:15.173445 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:44:15.173696 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43065
--builtin_ntp_servers=127.5.56.148:39289
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 02:44:15.175211 10367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.167525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43065 (local address 127.5.56.130:33303)
0925 02:44:15.168472 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:15.168535 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:15.170726 (+ 2191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:15.171675 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:15.171687 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:15.171819 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:15.172631 (+ 812us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:44:15.172634 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:15.172687 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:44:15.172808 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:15.173165 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:15.173176 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:15.173185 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:44:15.173486 (+ 301us) client_negotiation.cc:240] Negotiation successful
0925 02:44:15.173779 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":266,"threads_started":1}
I20260925 02:44:15.176683 10365 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43065
I20260925 02:44:15.177018 10365 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:15.177706 10365 heartbeater.cc:511] Master 127.5.56.190:43065 requested a full tablet report, sending...
I20260925 02:44:15.179248 10046 ts_manager.cc:194] Registered new tserver with Master: c9c72144132046c0be7869bf84719541 (127.5.56.130:41187)
W20260925 02:44:15.528306 10368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:15.528622 10368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:15.528774 10368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:15.536933 10368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:15.537055 10368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:15.537231 10368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:44:15.547071 10368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39289
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43065
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:15.548478 10368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:15.550367 10368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:15.564109 10373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:15.564123 10376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:15.564162 10374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:15.565055 10368 server_base.cc:1034] running on GCE node
I20260925 02:44:15.565729 10368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:15.566828 10368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:15.568082 10368 hybrid_clock.cc:648] HybridClock initialized: now 1790304255568012 us; error 54 us; skew 500 ppm
I20260925 02:44:15.568455 10368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:44:15.571125 10368 webserver.cc:460] Webserver started at http://127.5.56.131:44015/ using document root <none> and password file <none>
I20260925 02:44:15.571892 10368 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:15.572013 10368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:15.572366 10368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:15.575433 10368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "904f3050ca7d4edc91c187249d790071"
format_stamp: "Formatted at 2026-09-25 02:44:15 on dist-test-slave-n563"
server_key: "01bc9ef98b3fa44b430dedf42ff1eff3"
server_key_iv: "347a5c56c2fcad5c93e6939280ba7565"
server_key_version: "encryptionkey@0"
I20260925 02:44:15.576303 10368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "904f3050ca7d4edc91c187249d790071"
format_stamp: "Formatted at 2026-09-25 02:44:15 on dist-test-slave-n563"
server_key: "01bc9ef98b3fa44b430dedf42ff1eff3"
server_key_iv: "347a5c56c2fcad5c93e6939280ba7565"
server_key_version: "encryptionkey@0"
I20260925 02:44:15.582051 10368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:44:15.586475 10382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:15.588219 10368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:44:15.588454 10368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "904f3050ca7d4edc91c187249d790071"
format_stamp: "Formatted at 2026-09-25 02:44:15 on dist-test-slave-n563"
server_key: "01bc9ef98b3fa44b430dedf42ff1eff3"
server_key_iv: "347a5c56c2fcad5c93e6939280ba7565"
server_key_version: "encryptionkey@0"
I20260925 02:44:15.588729 10368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:15.595152 10368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:15.596092 10368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:15.596390 10368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:15.597430 10368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:15.599114 10368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:15.599231 10368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:15.599371 10368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:15.599481 10368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:15.634526 10368 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:36951
I20260925 02:44:15.634554 10497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:36951 every 8 connection(s)
I20260925 02:44:15.636124 10368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:44:15.636647 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 10368
I20260925 02:44:15.636818 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:44:15.637465 5346 external_mini_cluster.cc:1428] Setting key 2b96b4d3a1158e616927c7de05dbc5d9
I20260925 02:44:15.645112 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.639032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:56223 (local address 127.5.56.190:43065)
0925 02:44:15.639225 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:15.639232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:15.641106 (+ 1874us) server_negotiation.cc:407] Connection header received
0925 02:44:15.643001 (+ 1895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:15.643005 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:15.643063 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:15.643147 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:15.644472 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:15.644477 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:15.644480 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:15.644504 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:15.644538 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:15.644550 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:15.644650 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:15.644826 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 02:44:15.644878 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:44:15.646186 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.639690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43065 (local address 127.5.56.131:56223)
0925 02:44:15.640902 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:15.640972 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:15.642743 (+ 1771us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:15.643372 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:15.643382 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:15.643486 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:15.644159 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:44:15.644162 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:15.644204 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:44:15.644297 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:15.644636 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:15.644643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:15.644649 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:44:15.644879 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 02:44:15.645107 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":382,"threads_started":1}
I20260925 02:44:15.647706 10498 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43065
I20260925 02:44:15.648051 10498 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:15.648694 10498 heartbeater.cc:511] Master 127.5.56.190:43065 requested a full tablet report, sending...
I20260925 02:44:15.650250 10046 ts_manager.cc:194] Registered new tserver with Master: 904f3050ca7d4edc91c187249d790071 (127.5.56.131:36951)
I20260925 02:44:15.653692 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:44:15.661077 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.658490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.5.56.190:43065)
0925 02:44:15.658660 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:15.658665 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:15.659271 (+ 606us) server_negotiation.cc:407] Connection header received
0925 02:44:15.659433 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:15.659436 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:15.659493 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:15.659571 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:15.660477 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:15.660481 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:15.660484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:15.660501 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:15.660534 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:15.660547 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:15.660686 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:15.660781 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 02:44:15.660832 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:44:15.665349 10046 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53674:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:44:15.673060 10046 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:44:15.693255 10510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.686873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34471 (local address 127.0.0.1:43678)
0925 02:44:15.687648 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:15.687695 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:15.687872 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:15.691021 (+ 3149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:15.691028 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:15.691054 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:15.691751 (+ 697us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:44:15.691755 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:15.691787 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:44:15.691879 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:15.692439 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:15.692451 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:15.692460 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 02:44:15.692652 (+ 192us) client_negotiation.cc:240] Negotiation successful
0925 02:44:15.692760 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":363,"threads_started":1}
I20260925 02:44:15.693775 10511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.687062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:41187 (local address 127.0.0.1:43832)
0925 02:44:15.688765 (+ 1703us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:15.688783 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:15.688897 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:15.690534 (+ 1637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:15.690545 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:15.690643 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:15.691751 (+ 1108us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:44:15.691755 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:15.691787 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:44:15.691880 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:15.692998 (+ 1118us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:15.693003 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:15.693008 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:44:15.693138 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 02:44:15.693195 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1568,"thread_start_us":1410,"threads_started":1}
I20260925 02:44:15.694013 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.687406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43832 (local address 127.5.56.130:41187)
0925 02:44:15.688526 (+ 1120us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:15.688537 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:15.689022 (+ 485us) server_negotiation.cc:407] Connection header received
0925 02:44:15.689102 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:15.689110 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:15.689388 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:15.689557 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:15.692416 (+ 2859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:15.692434 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:15.692447 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:15.692476 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:15.692539 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:15.692564 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:15.693489 (+ 925us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:15.693584 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 02:44:15.693702 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":730,"threads_started":1}
I20260925 02:44:15.694224 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.687486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43678 (local address 127.5.56.129:34471)
0925 02:44:15.689602 (+ 2116us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:15.689611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:15.689642 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:44:15.689708 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:15.689715 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:15.689932 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:15.690086 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:15.692092 (+ 2006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:15.692104 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:15.692124 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:15.692156 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:15.692226 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:15.692251 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:15.692383 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:15.693740 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0925 02:44:15.693836 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1925,"thread_start_us":2045,"threads_started":1}
I20260925 02:44:15.698423 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.689784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.5.56.131:36951)
0925 02:44:15.693719 (+ 3935us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:15.693732 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:15.693771 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:44:15.693846 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:15.694014 (+ 168us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:15.695376 (+ 1362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:15.695567 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:15.697501 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:15.697510 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:15.697527 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:15.697549 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:15.697612 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:15.697634 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:15.697825 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:15.697890 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 02:44:15.697991 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3459,"thread_start_us":3268,"threads_started":1}
I20260925 02:44:15.698446 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.689395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:36951 (local address 127.0.0.1:51826)
0925 02:44:15.692754 (+ 3359us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:15.692771 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:15.692863 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:15.696402 (+ 3539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:15.696407 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:15.696449 (+ 42us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:15.697229 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:44:15.697235 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:15.697256 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:44:15.697308 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:15.697759 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:15.697765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:15.697770 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:44:15.697907 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 02:44:15.697969 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2858,"thread_start_us":2735,"threads_started":1}
I20260925 02:44:15.698221 10300 tablet_service.cc:1467] Processing CreateTablet for tablet 2086e2acb8be416ba997b04e19c24b58 (DEFAULT_TABLE table=test-table [id=be435410b2dd464a8e5f9462564fb158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:15.698223 10168 tablet_service.cc:1467] Processing CreateTablet for tablet 2086e2acb8be416ba997b04e19c24b58 (DEFAULT_TABLE table=test-table [id=be435410b2dd464a8e5f9462564fb158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:15.699970 10300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2086e2acb8be416ba997b04e19c24b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:15.700258 10168 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2086e2acb8be416ba997b04e19c24b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:15.702435 10431 tablet_service.cc:1467] Processing CreateTablet for tablet 2086e2acb8be416ba997b04e19c24b58 (DEFAULT_TABLE table=test-table [id=be435410b2dd464a8e5f9462564fb158]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:15.703945 10431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2086e2acb8be416ba997b04e19c24b58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:15.720965 10516 tablet_bootstrap.cc:492] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: Bootstrap starting.
I20260925 02:44:15.724519 10517 tablet_bootstrap.cc:492] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986: Bootstrap starting.
I20260925 02:44:15.729055 10517 tablet_bootstrap.cc:654] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:15.729558 10233 heartbeater.cc:503] Master 127.5.56.190:43065 was elected leader, sending a full tablet report...
I20260925 02:44:15.731082 10517 log.cc:824] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:15.731482 10518 tablet_bootstrap.cc:492] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: Bootstrap starting.
I20260925 02:44:15.734670 10516 tablet_bootstrap.cc:654] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:15.735838 10517 tablet_bootstrap.cc:492] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986: No bootstrap required, opened a new log
I20260925 02:44:15.736114 10516 log.cc:824] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:15.736325 10517 ts_tablet_manager.cc:1397] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.004s
I20260925 02:44:15.740571 10518 tablet_bootstrap.cc:654] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:15.742017 10518 log.cc:824] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:15.745211 10516 tablet_bootstrap.cc:492] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: No bootstrap required, opened a new log
I20260925 02:44:15.745545 10516 ts_tablet_manager.cc:1397] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.004s
I20260925 02:44:15.745993 10518 tablet_bootstrap.cc:492] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: No bootstrap required, opened a new log
I20260925 02:44:15.746273 10518 ts_tablet_manager.cc:1397] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260925 02:44:15.747570 10517 raft_consensus.cc:348] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d1b7db8df545ab8022be46fde2b986" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34471 } } peers { permanent_uuid: "c9c72144132046c0be7869bf84719541" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41187 } } peers { permanent_uuid: "904f3050ca7d4edc91c187249d790071" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36951 } }
I20260925 02:44:15.748067 10517 raft_consensus.cc:374] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:15.748167 10517 raft_consensus.cc:729] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35d1b7db8df545ab8022be46fde2b986, State: Initialized, Role: FOLLOWER
I20260925 02:44:15.748858 10517 consensus_queue.cc:260] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d1b7db8df545ab8022be46fde2b986" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34471 } } peers { permanent_uuid: "c9c72144132046c0be7869bf84719541" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41187 } } peers { permanent_uuid: "904f3050ca7d4edc91c187249d790071" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36951 } }
I20260925 02:44:15.750897 10517 ts_tablet_manager.cc:1428] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 02:44:15.756023 10516 raft_consensus.cc:348] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d1b7db8df545ab8022be46fde2b986" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34471 } } peers { permanent_uuid: "c9c72144132046c0be7869bf84719541" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41187 } } peers { permanent_uuid: "904f3050ca7d4edc91c187249d790071" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36951 } }
I20260925 02:44:15.756467 10516 raft_consensus.cc:374] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:15.756114 10518 raft_consensus.cc:348] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d1b7db8df545ab8022be46fde2b986" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34471 } } peers { permanent_uuid: "c9c72144132046c0be7869bf84719541" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41187 } } peers { permanent_uuid: "904f3050ca7d4edc91c187249d790071" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36951 } }
I20260925 02:44:15.756574 10516 raft_consensus.cc:729] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c72144132046c0be7869bf84719541, State: Initialized, Role: FOLLOWER
I20260925 02:44:15.756619 10518 raft_consensus.cc:374] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:15.756711 10518 raft_consensus.cc:729] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 904f3050ca7d4edc91c187249d790071, State: Initialized, Role: FOLLOWER
I20260925 02:44:15.757306 10518 consensus_queue.cc:260] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d1b7db8df545ab8022be46fde2b986" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34471 } } peers { permanent_uuid: "c9c72144132046c0be7869bf84719541" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41187 } } peers { permanent_uuid: "904f3050ca7d4edc91c187249d790071" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36951 } }
I20260925 02:44:15.757308 10516 consensus_queue.cc:260] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d1b7db8df545ab8022be46fde2b986" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34471 } } peers { permanent_uuid: "c9c72144132046c0be7869bf84719541" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41187 } } peers { permanent_uuid: "904f3050ca7d4edc91c187249d790071" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36951 } }
I20260925 02:44:15.758662 10365 heartbeater.cc:503] Master 127.5.56.190:43065 was elected leader, sending a full tablet report...
I20260925 02:44:15.759130 10498 heartbeater.cc:503] Master 127.5.56.190:43065 was elected leader, sending a full tablet report...
I20260925 02:44:15.759382 10516 ts_tablet_manager.cc:1428] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 02:44:15.760835 10518 ts_tablet_manager.cc:1428] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: Time spent starting tablet: real 0.014s user 0.007s sys 0.007s
W20260925 02:44:15.888159 10499 tablet.cc:2369] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:44:15.908357 10523 raft_consensus.cc:482] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:44:15.908716 10523 raft_consensus.cc:504] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d1b7db8df545ab8022be46fde2b986" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34471 } } peers { permanent_uuid: "c9c72144132046c0be7869bf84719541" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41187 } } peers { permanent_uuid: "904f3050ca7d4edc91c187249d790071" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36951 } }
I20260925 02:44:15.911056 10523 leader_election.cc:288] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35d1b7db8df545ab8022be46fde2b986 (127.5.56.129:34471), c9c72144132046c0be7869bf84719541 (127.5.56.130:41187)
I20260925 02:44:15.914196 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.911659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:53641 (local address 127.5.56.130:41187)
0925 02:44:15.911875 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:15.911881 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:15.911901 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:44:15.911967 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:15.911972 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:15.912032 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:15.912149 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:15.913407 (+ 1258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:15.913414 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:15.913418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:15.913440 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:15.913503 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:15.913523 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:15.913703 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:15.913819 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 02:44:15.913892 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:44:15.914235 10500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.911434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:41187 (local address 127.5.56.131:53641)
0925 02:44:15.911642 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:15.911659 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:15.911795 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:15.912337 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:15.912341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:15.912360 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:15.913117 (+ 757us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:44:15.913122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:15.913144 (+ 22us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:44:15.913200 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:15.913672 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:15.913677 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:15.913681 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 02:44:15.913826 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 02:44:15.913945 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 02:44:15.915279 10525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.911861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34471 (local address 127.5.56.131:36621)
0925 02:44:15.912671 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:15.912703 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:15.912856 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:15.913377 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:15.913382 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:15.913408 (+ 26us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:15.914188 (+ 780us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:44:15.914194 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:15.914220 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:44:15.914276 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:15.914804 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:15.914809 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:15.914812 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 02:44:15.914921 (+ 109us) client_negotiation.cc:240] Negotiation successful
0925 02:44:15.914982 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"thread_start_us":391,"threads_started":1}
I20260925 02:44:15.915303 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.912022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:36621 (local address 127.5.56.129:34471)
0925 02:44:15.912211 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:15.912217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:15.912822 (+ 605us) server_negotiation.cc:407] Connection header received
0925 02:44:15.913015 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:15.913022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:15.913091 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:15.913197 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:15.914491 (+ 1294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:15.914498 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:15.914503 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:15.914531 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:15.914592 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:15.914611 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:15.914708 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:15.914957 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 02:44:15.915032 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 02:44:15.915464 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2086e2acb8be416ba997b04e19c24b58" candidate_uuid: "904f3050ca7d4edc91c187249d790071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9c72144132046c0be7869bf84719541" is_pre_election: true
I20260925 02:44:15.916065 10320 raft_consensus.cc:2454] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 904f3050ca7d4edc91c187249d790071 in term 0.
W20260925 02:44:15.916435 10366 tablet.cc:2369] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:44:15.916235 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2086e2acb8be416ba997b04e19c24b58" candidate_uuid: "904f3050ca7d4edc91c187249d790071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35d1b7db8df545ab8022be46fde2b986" is_pre_election: true
I20260925 02:44:15.916846 10188 raft_consensus.cc:2454] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 904f3050ca7d4edc91c187249d790071 in term 0.
I20260925 02:44:15.916917 10387 leader_election.cc:302] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 904f3050ca7d4edc91c187249d790071, c9c72144132046c0be7869bf84719541; no voters:
I20260925 02:44:15.917357 10523 raft_consensus.cc:2781] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:44:15.917557 10523 raft_consensus.cc:482] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:44:15.917668 10523 raft_consensus.cc:3037] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:15.920198 10523 raft_consensus.cc:504] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d1b7db8df545ab8022be46fde2b986" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34471 } } peers { permanent_uuid: "c9c72144132046c0be7869bf84719541" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41187 } } peers { permanent_uuid: "904f3050ca7d4edc91c187249d790071" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36951 } }
I20260925 02:44:15.920925 10523 leader_election.cc:288] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [CANDIDATE]: Term 1 election: Requested vote from peers 35d1b7db8df545ab8022be46fde2b986 (127.5.56.129:34471), c9c72144132046c0be7869bf84719541 (127.5.56.130:41187)
I20260925 02:44:15.921478 10188 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2086e2acb8be416ba997b04e19c24b58" candidate_uuid: "904f3050ca7d4edc91c187249d790071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35d1b7db8df545ab8022be46fde2b986"
I20260925 02:44:15.921623 10320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2086e2acb8be416ba997b04e19c24b58" candidate_uuid: "904f3050ca7d4edc91c187249d790071" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9c72144132046c0be7869bf84719541"
I20260925 02:44:15.921712 10188 raft_consensus.cc:3037] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:15.921872 10320 raft_consensus.cc:3037] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:15.924160 10188 raft_consensus.cc:2454] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 904f3050ca7d4edc91c187249d790071 in term 1.
I20260925 02:44:15.924180 10320 raft_consensus.cc:2454] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 904f3050ca7d4edc91c187249d790071 in term 1.
I20260925 02:44:15.924700 10384 leader_election.cc:302] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35d1b7db8df545ab8022be46fde2b986, 904f3050ca7d4edc91c187249d790071; no voters:
I20260925 02:44:15.925040 10523 raft_consensus.cc:2781] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:15.925529 10523 raft_consensus.cc:686] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 1 LEADER]: Becoming Leader. State: Replica: 904f3050ca7d4edc91c187249d790071, State: Running, Role: LEADER
I20260925 02:44:15.926030 10523 consensus_queue.cc:237] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d1b7db8df545ab8022be46fde2b986" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34471 } } peers { permanent_uuid: "c9c72144132046c0be7869bf84719541" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41187 } } peers { permanent_uuid: "904f3050ca7d4edc91c187249d790071" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36951 } }
I20260925 02:44:15.930922 10045 catalog_manager.cc:5495] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 reported cstate change: term changed from 0 to 1, leader changed from <none> to 904f3050ca7d4edc91c187249d790071 (127.5.56.131). New cstate: current_term: 1 leader_uuid: "904f3050ca7d4edc91c187249d790071" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35d1b7db8df545ab8022be46fde2b986" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c9c72144132046c0be7869bf84719541" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "904f3050ca7d4edc91c187249d790071" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36951 } health_report { overall_health: HEALTHY } } }
I20260925 02:44:15.949110 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.947054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.5.56.131:36951)
0925 02:44:15.947240 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:15.947244 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:15.947384 (+ 140us) server_negotiation.cc:407] Connection header received
0925 02:44:15.947576 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:15.947580 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:15.947663 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:15.947755 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:15.948616 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:15.948621 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:15.948624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:15.948639 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:15.948673 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:15.948685 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:15.948790 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:15.948858 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 02:44:15.948905 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:44:15.958158 10188 raft_consensus.cc:1261] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986 [term 1 FOLLOWER]: Refusing update from remote peer 904f3050ca7d4edc91c187249d790071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:15.958170 10320 raft_consensus.cc:1261] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541 [term 1 FOLLOWER]: Refusing update from remote peer 904f3050ca7d4edc91c187249d790071: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:15.959306 10526 consensus_queue.cc:1035] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35d1b7db8df545ab8022be46fde2b986" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 34471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:15.959707 10523 consensus_queue.cc:1035] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9c72144132046c0be7869bf84719541" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:44:15.963733 10234 tablet.cc:2369] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:44:15.970026 10529 mvcc.cc:205] Tried to move back new op lower bound from 7333086232393564160 to 7333086232277045248. Current Snapshot: MvccSnapshot[applied={T|T < 7333086232393564160}]
I20260925 02:44:15.972064 10530 mvcc.cc:205] Tried to move back new op lower bound from 7333086232393564160 to 7333086232277045248. Current Snapshot: MvccSnapshot[applied={T|T < 7333086232393564160}]
I20260925 02:44:15.987995 10045 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:15.988369 10045 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53674:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:15.992641 10045 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c9b1d634208484ab077b7eafa4feaf4: Sending DeleteTablet for 3 replicas of tablet 2086e2acb8be416ba997b04e19c24b58
I20260925 02:44:15.994072 10168 tablet_service.cc:1514] Processing DeleteTablet for tablet 2086e2acb8be416ba997b04e19c24b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:15 UTC) from {username='slave'} at 127.0.0.1:43678
I20260925 02:44:15.994063 10300 tablet_service.cc:1514] Processing DeleteTablet for tablet 2086e2acb8be416ba997b04e19c24b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:15 UTC) from {username='slave'} at 127.0.0.1:43832
I20260925 02:44:15.994638 10431 tablet_service.cc:1514] Processing DeleteTablet for tablet 2086e2acb8be416ba997b04e19c24b58 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:15 UTC) from {username='slave'} at 127.0.0.1:51826
I20260925 02:44:15.996635 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 10105
I20260925 02:44:16.008666 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:15.998247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.5.56.129:34471)
0925 02:44:15.998467 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:15.998482 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:16.003110 (+ 4628us) server_negotiation.cc:407] Connection header received
0925 02:44:16.004335 (+ 1225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:16.004345 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:16.004509 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:16.004612 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:16.007335 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:16.007347 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:16.007355 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:16.007388 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:16.007447 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:16.007469 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:16.007617 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:16.008130 (+ 513us) server_negotiation.cc:299] Negotiation successful
0925 02:44:16.008226 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:44:16.010082 10228 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:16.011581 10542 tablet_replica.cc:330] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: stopping tablet replica
I20260925 02:44:16.012075 10542 raft_consensus.cc:2229] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:44:16.012746 10542 raft_consensus.cc:2258] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:16.016971 10540 tablet_replica.cc:330] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: stopping tablet replica
I20260925 02:44:16.017644 10540 raft_consensus.cc:2229] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:16.018042 10542 ts_tablet_manager.cc:1905] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:16.018417 10540 raft_consensus.cc:2258] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:16.022363 10540 ts_tablet_manager.cc:1905] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:16.026707 10542 ts_tablet_manager.cc:1918] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:16.026917 10542 log.cc:1196] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/2086e2acb8be416ba997b04e19c24b58
I20260925 02:44:16.027531 10542 ts_tablet_manager.cc:1939] T 2086e2acb8be416ba997b04e19c24b58 P 904f3050ca7d4edc91c187249d790071: Deleting consensus metadata
I20260925 02:44:16.029753 10033 catalog_manager.cc:4841] TS 904f3050ca7d4edc91c187249d790071 (127.5.56.131:36951): tablet 2086e2acb8be416ba997b04e19c24b58 (table test-table [id=be435410b2dd464a8e5f9462564fb158]) successfully deleted
I20260925 02:44:16.029794 10540 ts_tablet_manager.cc:1918] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:16.029991 10540 log.cc:1196] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/2086e2acb8be416ba997b04e19c24b58
I20260925 02:44:16.030570 10540 ts_tablet_manager.cc:1939] T 2086e2acb8be416ba997b04e19c24b58 P c9c72144132046c0be7869bf84719541: Deleting consensus metadata
I20260925 02:44:16.032544 10033 catalog_manager.cc:4841] TS c9c72144132046c0be7869bf84719541 (127.5.56.130:41187): tablet 2086e2acb8be416ba997b04e19c24b58 (table test-table [id=be435410b2dd464a8e5f9462564fb158]) successfully deleted
I20260925 02:44:16.122121 10541 tablet_replica.cc:330] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986: stopping tablet replica
I20260925 02:44:16.122677 10541 raft_consensus.cc:2229] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:16.123202 10541 raft_consensus.cc:2258] T 2086e2acb8be416ba997b04e19c24b58 P 35d1b7db8df545ab8022be46fde2b986 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:16.124330 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 10105
W20260925 02:44:16.142019 10030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.56.129:34471 (error 108)
I20260925 02:44:16.142858 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 10237
I20260925 02:44:16.145355 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:16.144724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34471 (local address 127.0.0.1:43702)
0925 02:44:16.145006 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:16.145170 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
W20260925 02:44:16.145661 10030 catalog_manager.cc:4584] TS 35d1b7db8df545ab8022be46fde2b986 (127.5.56.129:34471): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2086e2acb8be416ba997b04e19c24b58: Network error: Client connection negotiation failed: client connection to 127.5.56.129:34471: connect: Connection refused (error 111)
I20260925 02:44:16.146802 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:16.144158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43834 (local address 127.5.56.130:41187)
0925 02:44:16.144357 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:16.144364 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:16.144387 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:44:16.144456 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:16.144461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:16.144573 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:16.144676 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:16.146075 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:16.146080 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:16.146084 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:16.146104 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:16.146141 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:16.146156 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:16.146281 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:16.146468 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 02:44:16.146536 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:44:16.147691 10360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:16.199419 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:16.198841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34471 (local address 127.0.0.1:43716)
0925 02:44:16.199093 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:16.199226 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 02:44:16.200646 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:16.200213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34471 (local address 127.0.0.1:43720)
0925 02:44:16.200389 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:16.200503 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 02:44:16.239526 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 10237
I20260925 02:44:16.258498 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 10368
I20260925 02:44:16.261729 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:16.261108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34471 (local address 127.0.0.1:43732)
0925 02:44:16.261394 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:16.261509 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260925 02:44:16.261870 10515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:16.259334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51834 (local address 127.5.56.131:36951)
0925 02:44:16.259508 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:16.259515 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:16.259553 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:44:16.259794 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:16.259802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:16.259856 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:16.260033 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:16.260976 (+ 943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:16.260982 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:16.260986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:16.261003 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:16.261042 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:16.261053 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:16.261234 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:16.261560 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 02:44:16.261616 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:44:16.262907 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:16.262549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34471 (local address 127.0.0.1:43740)
0925 02:44:16.262720 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:16.262799 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 02:44:16.263039 10493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:16.345505 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:16.344899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34471 (local address 127.0.0.1:43754)
0925 02:44:16.345168 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:16.345287 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 02:44:16.346697 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:16.346221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34471 (local address 127.0.0.1:43764)
0925 02:44:16.346458 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:16.346552 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260925 02:44:16.391149 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 10368
I20260925 02:44:16.411543 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 10015
I20260925 02:44:16.412776 10076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:16.523483 10514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:16.522711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:34471 (local address 127.0.0.1:43772)
0925 02:44:16.523030 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:16.523161 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:34471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260925 02:44:16.523684 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 10015
2026-09-25T02:44:16Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2905 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 02:44:16.568508 5346 mini_oidc.cc:152] Starting JWKS server
I20260925 02:44:16.573225 5346 webserver.cc:460] Webserver started at https://127.0.0.1:34431/ using document root <none> and password file <none>
I20260925 02:44:16.573541 5346 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:44:16.574213 5346 webserver.cc:460] Webserver started at http://127.0.0.1:33567/ using document root <none> and password file <none>
2026-09-25T02:44:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:16Z Disabled control of system clock
I20260925 02:44:16.605203 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:39309
--webserver_interface=127.5.56.190
--webserver_port=0
--builtin_ntp_servers=127.5.56.148:36543
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:39309
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34431/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/testchainca.cert with env {}
W20260925 02:44:16.957624 10554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:16.957968 10554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:16.958060 10554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:16.965731 10554 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:16.965955 10554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:16.966050 10554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:16.966121 10554 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:16.966164 10554 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:44:16.966228 10554 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:44:16.966266 10554 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34431/jwks.json
I20260925 02:44:16.977178 10554 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36543
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:39309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:39309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34431/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:16.978612 10554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:16.980000 10554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:16.993434 10559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:16.993539 10560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:16.993491 10562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:16.994539 10554 server_base.cc:1034] running on GCE node
I20260925 02:44:16.995452 10554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:16.997107 10554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:16.998353 10554 hybrid_clock.cc:648] HybridClock initialized: now 1790304256998298 us; error 44 us; skew 500 ppm
I20260925 02:44:16.998852 10554 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:44:17.001554 10554 webserver.cc:460] Webserver started at http://127.5.56.190:46245/ using document root <none> and password file <none>
I20260925 02:44:17.002465 10554 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:17.002631 10554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:17.002993 10554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:17.006495 10554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "207afe004b914fdf9a8b5c5cfd2c2c8d"
format_stamp: "Formatted at 2026-09-25 02:44:17 on dist-test-slave-n563"
server_key: "4d3a8cd69662ca81c843096f1e58e6c5"
server_key_iv: "3a7d683e410c1e491232a584593db9cc"
server_key_version: "encryptionkey@0"
I20260925 02:44:17.007467 10554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "207afe004b914fdf9a8b5c5cfd2c2c8d"
format_stamp: "Formatted at 2026-09-25 02:44:17 on dist-test-slave-n563"
server_key: "4d3a8cd69662ca81c843096f1e58e6c5"
server_key_iv: "3a7d683e410c1e491232a584593db9cc"
server_key_version: "encryptionkey@0"
I20260925 02:44:17.013768 10554 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:44:17.017800 10568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:17.019480 10554 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:44:17.019716 10554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "207afe004b914fdf9a8b5c5cfd2c2c8d"
format_stamp: "Formatted at 2026-09-25 02:44:17 on dist-test-slave-n563"
server_key: "4d3a8cd69662ca81c843096f1e58e6c5"
server_key_iv: "3a7d683e410c1e491232a584593db9cc"
server_key_version: "encryptionkey@0"
I20260925 02:44:17.019972 10554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:17.054301 10554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:17.055305 10554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:17.055666 10554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:17.075578 10554 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:39309
I20260925 02:44:17.075596 10620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:39309 every 8 connection(s)
I20260925 02:44:17.077127 10554 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:17.078552 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 10554
I20260925 02:44:17.078778 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:17.079602 5346 external_mini_cluster.cc:1428] Setting key 6710a6fcbc48e0abe26923453472ccef
I20260925 02:44:17.082330 10621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:17.095504 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:17.081749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.5.56.190:39309)
0925 02:44:17.082874 (+ 1125us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:17.082886 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:17.082933 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:44:17.084630 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:17.084653 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:17.085218 (+ 565us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:17.085607 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:17.086458 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:17.087946 (+ 1488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:17.090025 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:17.090981 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:17.091139 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:17.092597 (+ 1458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:17.092608 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:17.092626 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:17.093316 (+ 690us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:17.093424 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:17.093453 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:17.093552 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:17.093901 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 02:44:17.094316 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":395,"threads_started":1}
I20260925 02:44:17.095930 10621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d: Bootstrap starting.
I20260925 02:44:17.099481 10621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:17.099874 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:44:17.100900 10621 log.cc:824] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:17.104209 10621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d: No bootstrap required, opened a new log
I20260925 02:44:17.111881 10621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207afe004b914fdf9a8b5c5cfd2c2c8d" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39309 } }
I20260925 02:44:17.112301 10621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:17.112385 10621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 207afe004b914fdf9a8b5c5cfd2c2c8d, State: Initialized, Role: FOLLOWER
I20260925 02:44:17.113021 10621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207afe004b914fdf9a8b5c5cfd2c2c8d" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39309 } }
I20260925 02:44:17.113255 10621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:17.113358 10621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:17.113504 10621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:17.115837 10621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207afe004b914fdf9a8b5c5cfd2c2c8d" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39309 } }
I20260925 02:44:17.116448 10621 leader_election.cc:302] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 207afe004b914fdf9a8b5c5cfd2c2c8d; no voters:
I20260925 02:44:17.117161 10621 leader_election.cc:288] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:17.117348 10626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:17.118055 10626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [term 1 LEADER]: Becoming Leader. State: Replica: 207afe004b914fdf9a8b5c5cfd2c2c8d, State: Running, Role: LEADER
I20260925 02:44:17.118680 10626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207afe004b914fdf9a8b5c5cfd2c2c8d" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39309 } }
I20260925 02:44:17.119482 10621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:17.121963 10628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 207afe004b914fdf9a8b5c5cfd2c2c8d. Latest consensus state: current_term: 1 leader_uuid: "207afe004b914fdf9a8b5c5cfd2c2c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207afe004b914fdf9a8b5c5cfd2c2c8d" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39309 } } }
I20260925 02:44:17.122123 10627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "207afe004b914fdf9a8b5c5cfd2c2c8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "207afe004b914fdf9a8b5c5cfd2c2c8d" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39309 } } }
I20260925 02:44:17.122454 10628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:17.122532 10627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:17.125173 10633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:17.131289 10633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:17.142091 10633 catalog_manager.cc:1347] Generated new cluster ID: b502c9e423d94d7d8912105631e64d95
I20260925 02:44:17.142249 10633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:17.167088 10633 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:17.168608 10633 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:17.184943 10633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 207afe004b914fdf9a8b5c5cfd2c2c8d: Generated new TSK 0
I20260925 02:44:17.186017 10633 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:44:17.239600 10651 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 02:44:17.239871 10651 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:44:17.242053 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:17.212990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.5.56.190:39309)
0925 02:44:17.213225 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:17.213232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:17.213951 (+ 719us) server_negotiation.cc:407] Connection header received
0925 02:44:17.214540 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:17.214545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:17.214626 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:17.214753 (+ 127us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:44:17.215424 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:17.216173 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:17.218454 (+ 2281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:17.218949 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:17.219055 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:17.219839 (+ 784us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:44:17.241433 (+ 21594us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:44:17.241785 (+ 352us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50688: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":78}
W20260925 02:44:17.242228 10624 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50688: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 02:44:17.244823 10650 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:17.212932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:39309 (local address 127.0.0.1:50688)
0925 02:44:17.213827 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:17.213850 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:17.214151 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:17.214824 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:17.214830 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:17.214844 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:44:17.215262 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:17.215274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:17.216336 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:17.216342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:17.218211 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:17.218229 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:17.219132 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:17.219139 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:17.219373 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:17.219480 (+ 107us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:44:17.241821 (+ 22341us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 02:44:17.242270 (+ 449us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.56.190:39309: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":509,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.5.56.190:39309: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34431/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 02:44:17.247270 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 10554
I20260925 02:44:17.248582 10616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:17.349386 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 10554
2026-09-25T02:44:17Z chronyd exiting
I20260925 02:44:17.401687 5346 test_util.cc:175] -----------------------------------------------
I20260925 02:44:17.401885 5346 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790304212743187-5346-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (839 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 02:44:17.405937 5346 mini_oidc.cc:152] Starting JWKS server
I20260925 02:44:17.408077 5346 webserver.cc:460] Webserver started at https://127.0.0.1:40341/ using document root <none> and password file <none>
I20260925 02:44:17.408355 5346 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:44:17.409106 5346 webserver.cc:460] Webserver started at http://127.0.0.1:43951/ using document root <none> and password file <none>
2026-09-25T02:44:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:17Z Disabled control of system clock
I20260925 02:44:17.440313 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:44019
--webserver_interface=127.5.56.190
--webserver_port=0
--builtin_ntp_servers=127.5.56.148:38531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:44019
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40341/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/testchainca.cert with env {}
W20260925 02:44:17.785669 10658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:17.785998 10658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:17.786087 10658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:17.793787 10658 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:17.793905 10658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:17.793942 10658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:17.793996 10658 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:17.794040 10658 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:44:17.794098 10658 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:44:17.794137 10658 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40341/jwks.json
I20260925 02:44:17.804761 10658 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:38531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:44019
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:44019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40341/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:17.806154 10658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:17.807554 10658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:17.820178 10664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:17.820281 10663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:17.820299 10666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:17.821164 10658 server_base.cc:1034] running on GCE node
I20260925 02:44:17.822042 10658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:17.823711 10658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:17.825011 10658 hybrid_clock.cc:648] HybridClock initialized: now 1790304257824987 us; error 71 us; skew 500 ppm
I20260925 02:44:17.825439 10658 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:44:17.827932 10658 webserver.cc:460] Webserver started at http://127.5.56.190:35331/ using document root <none> and password file <none>
I20260925 02:44:17.828699 10658 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:17.828840 10658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:17.829191 10658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:17.832329 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "1ebc4d145a1a48489c490cee6d637231"
format_stamp: "Formatted at 2026-09-25 02:44:17 on dist-test-slave-n563"
server_key: "c13333b18295cbf3564bc13ec5c81222"
server_key_iv: "9bf862a7c9fabdea9aee929141461650"
server_key_version: "encryptionkey@0"
I20260925 02:44:17.833168 10658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "1ebc4d145a1a48489c490cee6d637231"
format_stamp: "Formatted at 2026-09-25 02:44:17 on dist-test-slave-n563"
server_key: "c13333b18295cbf3564bc13ec5c81222"
server_key_iv: "9bf862a7c9fabdea9aee929141461650"
server_key_version: "encryptionkey@0"
I20260925 02:44:17.838838 10658 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:44:17.842701 10672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:17.844213 10658 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:44:17.844439 10658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "1ebc4d145a1a48489c490cee6d637231"
format_stamp: "Formatted at 2026-09-25 02:44:17 on dist-test-slave-n563"
server_key: "c13333b18295cbf3564bc13ec5c81222"
server_key_iv: "9bf862a7c9fabdea9aee929141461650"
server_key_version: "encryptionkey@0"
I20260925 02:44:17.844689 10658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:17.868667 10658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:17.869663 10658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:17.870030 10658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:17.889747 10658 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:44019
I20260925 02:44:17.889775 10723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:44019 every 8 connection(s)
I20260925 02:44:17.891331 10658 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:17.892313 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 10658
I20260925 02:44:17.892551 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:17.893491 5346 external_mini_cluster.cc:1428] Setting key eb19199ba8bfe1d97c61eb14efe23808
I20260925 02:44:17.896751 10724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:17.907831 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:17.895687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.5.56.190:44019)
0925 02:44:17.896769 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:17.896782 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:17.896861 (+ 79us) server_negotiation.cc:407] Connection header received
0925 02:44:17.898636 (+ 1775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:17.898659 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:17.899148 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:17.899609 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:17.900233 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:17.901597 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:17.903055 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:17.903776 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:17.904001 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:17.905265 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:17.905276 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:17.905292 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:17.905900 (+ 608us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:17.905987 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:17.906009 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:17.906158 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:17.906447 (+ 289us) server_negotiation.cc:299] Negotiation successful
0925 02:44:17.906798 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":390,"threads_started":1}
I20260925 02:44:17.909248 10724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231: Bootstrap starting.
I20260925 02:44:17.911994 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:44:17.913143 10724 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:17.914480 10724 log.cc:824] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:17.917682 10724 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231: No bootstrap required, opened a new log
I20260925 02:44:17.926963 10724 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebc4d145a1a48489c490cee6d637231" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44019 } }
I20260925 02:44:17.927412 10724 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:17.927498 10724 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ebc4d145a1a48489c490cee6d637231, State: Initialized, Role: FOLLOWER
I20260925 02:44:17.928155 10724 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebc4d145a1a48489c490cee6d637231" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44019 } }
I20260925 02:44:17.928450 10724 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:17.928576 10724 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:17.928714 10724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:17.931734 10724 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebc4d145a1a48489c490cee6d637231" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44019 } }
I20260925 02:44:17.932291 10724 leader_election.cc:302] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ebc4d145a1a48489c490cee6d637231; no voters:
I20260925 02:44:17.932897 10724 leader_election.cc:288] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:17.933054 10729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:17.933853 10729 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [term 1 LEADER]: Becoming Leader. State: Replica: 1ebc4d145a1a48489c490cee6d637231, State: Running, Role: LEADER
I20260925 02:44:17.934530 10729 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebc4d145a1a48489c490cee6d637231" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44019 } }
I20260925 02:44:17.935410 10724 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:17.937599 10731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ebc4d145a1a48489c490cee6d637231. Latest consensus state: current_term: 1 leader_uuid: "1ebc4d145a1a48489c490cee6d637231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebc4d145a1a48489c490cee6d637231" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44019 } } }
I20260925 02:44:17.938191 10731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:17.937822 10730 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ebc4d145a1a48489c490cee6d637231" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ebc4d145a1a48489c490cee6d637231" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44019 } } }
I20260925 02:44:17.939661 10730 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:17.943188 10737 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:17.947801 10737 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:17.956696 10737 catalog_manager.cc:1347] Generated new cluster ID: 8d7f3c3b44ed43b98c17f20ddc9d159a
I20260925 02:44:17.956844 10737 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:17.974145 10737 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:17.976117 10737 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:17.983397 10737 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ebc4d145a1a48489c490cee6d637231: Generated new TSK 0
I20260925 02:44:17.984620 10737 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 02:44:18.033684 10755 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 02:44:18.033828 10755 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:44:18.034921 10727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:18.006429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.5.56.190:44019)
0925 02:44:18.006619 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:18.006624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:18.007047 (+ 423us) server_negotiation.cc:407] Connection header received
0925 02:44:18.007295 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:18.007299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:18.007381 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:18.007531 (+ 150us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:44:18.008082 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:18.008896 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:18.011052 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:18.011523 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:18.011620 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:18.012261 (+ 641us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:44:18.034245 (+ 21984us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:44:18.034674 (+ 429us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46812: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 02:44:18.035085 10727 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46812: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 02:44:18.035315 10754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:18.006214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:44019 (local address 127.0.0.1:46812)
0925 02:44:18.006919 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:18.006945 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:18.007111 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:18.007578 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:18.007585 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:18.007601 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:44:18.007904 (+ 303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:18.007914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:18.009050 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:18.009055 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:18.010901 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:18.010918 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:18.011716 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:18.011723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:18.011934 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:18.011987 (+ 53us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:44:18.034543 (+ 22556us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 02:44:18.034862 (+ 319us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.56.190:44019: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40341/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":403,"threads_started":1}
I20260925 02:44:18.036506 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 10658
I20260925 02:44:18.037772 10719 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:18.133387 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 10658
2026-09-25T02:44:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (784 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 02:44:18.190284 5346 mini_oidc.cc:152] Starting JWKS server
I20260925 02:44:18.192308 5346 webserver.cc:460] Webserver started at https://127.0.0.1:39501/ using document root <none> and password file <none>
I20260925 02:44:18.192607 5346 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 02:44:18.193321 5346 webserver.cc:460] Webserver started at http://127.0.0.1:35111/ using document root <none> and password file <none>
2026-09-25T02:44:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:18Z Disabled control of system clock
I20260925 02:44:18.223903 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:33911
--webserver_interface=127.5.56.190
--webserver_port=0
--builtin_ntp_servers=127.5.56.148:33879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:33911
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39501/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:44:18.574189 10762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:18.574649 10762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:18.574811 10762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:18.582511 10762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:18.582674 10762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:18.582773 10762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:18.582870 10762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:18.582957 10762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:44:18.583050 10762 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:44:18.583132 10762 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39501/jwks.json
I20260925 02:44:18.593052 10762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:33911
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:33911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39501/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:18.594525 10762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:18.596542 10762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:18.609588 10767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:18.609537 10768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:18.610076 10770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:18.610325 10762 server_base.cc:1034] running on GCE node
I20260925 02:44:18.611369 10762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:18.612926 10762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:18.614197 10762 hybrid_clock.cc:648] HybridClock initialized: now 1790304258614149 us; error 46 us; skew 500 ppm
I20260925 02:44:18.614655 10762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:44:18.617061 10762 webserver.cc:460] Webserver started at http://127.5.56.190:44973/ using document root <none> and password file <none>
I20260925 02:44:18.617841 10762 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:18.617965 10762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:18.618327 10762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:18.621449 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "29cd1b7809524ce49139fb3f7a35238b"
format_stamp: "Formatted at 2026-09-25 02:44:18 on dist-test-slave-n563"
server_key: "76b1970dd86897c32c44a0f59de014fa"
server_key_iv: "ea941ff0e9d870034d1e13a2da770787"
server_key_version: "encryptionkey@0"
I20260925 02:44:18.622339 10762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "29cd1b7809524ce49139fb3f7a35238b"
format_stamp: "Formatted at 2026-09-25 02:44:18 on dist-test-slave-n563"
server_key: "76b1970dd86897c32c44a0f59de014fa"
server_key_iv: "ea941ff0e9d870034d1e13a2da770787"
server_key_version: "encryptionkey@0"
I20260925 02:44:18.627854 10762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:44:18.631544 10776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:18.633005 10762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:44:18.633208 10762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "29cd1b7809524ce49139fb3f7a35238b"
format_stamp: "Formatted at 2026-09-25 02:44:18 on dist-test-slave-n563"
server_key: "76b1970dd86897c32c44a0f59de014fa"
server_key_iv: "ea941ff0e9d870034d1e13a2da770787"
server_key_version: "encryptionkey@0"
I20260925 02:44:18.633448 10762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:18.673949 10762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:18.674954 10762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:18.675294 10762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:18.694526 10762 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:33911
I20260925 02:44:18.694509 10827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:33911 every 8 connection(s)
I20260925 02:44:18.696100 10762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:18.700098 10828 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:18.706323 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 10762
I20260925 02:44:18.706588 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:18.707396 5346 external_mini_cluster.cc:1428] Setting key 5c9bbd27f242bde9066e8adfb7ca3ed0
I20260925 02:44:18.712353 10828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b: Bootstrap starting.
I20260925 02:44:18.715654 10828 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:18.716704 10828 log.cc:824] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:18.719885 10828 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b: No bootstrap required, opened a new log
I20260925 02:44:18.721421 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:18.709568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.5.56.190:33911)
0925 02:44:18.710700 (+ 1132us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:18.710712 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:18.710758 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:44:18.712492 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:18.712529 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:18.713017 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:18.713418 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:18.714013 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:18.715390 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:18.716792 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:18.717577 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:18.717794 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:18.719031 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:18.719039 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:18.719061 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:18.719619 (+ 558us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:18.719700 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:18.719721 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:18.719867 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:18.720110 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 02:44:18.720450 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":336,"threads_started":1}
I20260925 02:44:18.725651 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:44:18.728781 10828 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29cd1b7809524ce49139fb3f7a35238b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33911 } }
I20260925 02:44:18.729295 10828 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:18.729408 10828 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29cd1b7809524ce49139fb3f7a35238b, State: Initialized, Role: FOLLOWER
I20260925 02:44:18.730257 10828 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29cd1b7809524ce49139fb3f7a35238b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33911 } }
I20260925 02:44:18.730746 10828 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:18.730919 10828 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:18.731106 10828 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:18.733661 10828 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29cd1b7809524ce49139fb3f7a35238b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33911 } }
I20260925 02:44:18.734244 10828 leader_election.cc:302] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29cd1b7809524ce49139fb3f7a35238b; no voters:
I20260925 02:44:18.734902 10828 leader_election.cc:288] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:18.735044 10833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:18.735793 10833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [term 1 LEADER]: Becoming Leader. State: Replica: 29cd1b7809524ce49139fb3f7a35238b, State: Running, Role: LEADER
I20260925 02:44:18.736397 10833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29cd1b7809524ce49139fb3f7a35238b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33911 } }
I20260925 02:44:18.737350 10828 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:18.739671 10834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29cd1b7809524ce49139fb3f7a35238b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29cd1b7809524ce49139fb3f7a35238b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33911 } } }
I20260925 02:44:18.740142 10834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:18.739737 10835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29cd1b7809524ce49139fb3f7a35238b. Latest consensus state: current_term: 1 leader_uuid: "29cd1b7809524ce49139fb3f7a35238b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29cd1b7809524ce49139fb3f7a35238b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33911 } } }
I20260925 02:44:18.740484 10835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:18.743045 10839 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:18.748509 10839 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:18.758255 10839 catalog_manager.cc:1347] Generated new cluster ID: b9bd325e7cff403495c35d4b87e306b6
I20260925 02:44:18.758423 10839 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:18.783742 10839 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:18.785792 10839 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:18.805053 10839 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29cd1b7809524ce49139fb3f7a35238b: Generated new TSK 0
I20260925 02:44:18.806252 10839 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:44:18.834590 10831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:18.825456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.5.56.190:33911)
0925 02:44:18.825718 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:18.825728 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:18.826347 (+ 619us) server_negotiation.cc:407] Connection header received
0925 02:44:18.826650 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:18.826656 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:18.826750 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:18.826951 (+ 201us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:44:18.827400 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:18.828591 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:18.829977 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:18.830713 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:18.830851 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:18.831371 (+ 520us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:44:18.833833 (+ 2462us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:44:18.834190 (+ 357us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53114: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 02:44:18.834729 10831 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53114: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 02:44:18.836619 10857 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:18.825274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:33911 (local address 127.0.0.1:53114)
0925 02:44:18.826220 (+ 946us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:18.826237 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:18.826368 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:18.826968 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:18.826972 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:18.826982 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:44:18.827243 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:18.827251 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:18.828739 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:18.828743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:18.829829 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:18.829840 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:18.830901 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:18.830905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:18.831047 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:18.831090 (+ 43us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:44:18.834075 (+ 2985us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 02:44:18.836116 (+ 2041us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.56.190:33911: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39501/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":793,"mutex_wait_us":1672,"thread_start_us":642,"threads_started":1}
I20260925 02:44:18.838286 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 10762
I20260925 02:44:18.839591 10823 generic_service.cc:196] Checking for leaks (request via RPC)
E20260925 02:44:18.848965 10858 webserver.cc:533] Webserver: sslize failed:
E20260925 02:44:18.849072 10858 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 02:44:18.939355 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 10762
2026-09-25T02:44:18Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (806 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T02:44:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:19Z Disabled control of system clock
I20260925 02:44:19.026516 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43905
--webserver_interface=127.5.56.190
--webserver_port=0
--builtin_ntp_servers=127.5.56.148:46783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:43905
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/testchainca.cert with env {}
W20260925 02:44:19.377151 10863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:19.377523 10863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:19.377714 10863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:19.385756 10863 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:19.385926 10863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:19.386024 10863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:19.386126 10863 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:19.386251 10863 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 02:44:19.386366 10863 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 02:44:19.386469 10863 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 02:44:19.397666 10863 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:46783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:43905
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:19.399691 10863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:19.401630 10863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:19.416394 10868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:19.416339 10869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:19.416812 10863 server_base.cc:1034] running on GCE node
W20260925 02:44:19.416373 10871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:19.418036 10863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:19.419731 10863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:19.420993 10863 hybrid_clock.cc:648] HybridClock initialized: now 1790304259420948 us; error 32 us; skew 500 ppm
I20260925 02:44:19.421402 10863 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:44:19.423990 10863 webserver.cc:460] Webserver started at http://127.5.56.190:34355/ using document root <none> and password file <none>
I20260925 02:44:19.424757 10863 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:19.424899 10863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:19.425268 10863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:19.428380 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "d2d46fe45d1c4ef9ba63f222f89b9586"
format_stamp: "Formatted at 2026-09-25 02:44:19 on dist-test-slave-n563"
server_key: "74ef54fba12d96f3ba5f214a12720eda"
server_key_iv: "7b853967ae9515dc7763c76d8659b414"
server_key_version: "encryptionkey@0"
I20260925 02:44:19.429251 10863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "d2d46fe45d1c4ef9ba63f222f89b9586"
format_stamp: "Formatted at 2026-09-25 02:44:19 on dist-test-slave-n563"
server_key: "74ef54fba12d96f3ba5f214a12720eda"
server_key_iv: "7b853967ae9515dc7763c76d8659b414"
server_key_version: "encryptionkey@0"
I20260925 02:44:19.435150 10863 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:44:19.438827 10877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:19.440310 10863 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:44:19.440557 10863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "d2d46fe45d1c4ef9ba63f222f89b9586"
format_stamp: "Formatted at 2026-09-25 02:44:19 on dist-test-slave-n563"
server_key: "74ef54fba12d96f3ba5f214a12720eda"
server_key_iv: "7b853967ae9515dc7763c76d8659b414"
server_key_version: "encryptionkey@0"
I20260925 02:44:19.440809 10863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:19.464730 10863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:19.465811 10863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:19.466198 10863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:19.488318 10863 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:43905
I20260925 02:44:19.488368 10929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:43905 every 8 connection(s)
I20260925 02:44:19.490129 10863 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:19.494918 10930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:19.499007 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 10863
I20260925 02:44:19.499250 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:19.499905 5346 external_mini_cluster.cc:1428] Setting key 5ec57ed18b07bcd990750b60385824f0
I20260925 02:44:19.508915 10930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586: Bootstrap starting.
I20260925 02:44:19.513329 10930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:19.514931 10930 log.cc:824] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:19.518095 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:19.502194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.5.56.190:43905)
0925 02:44:19.503479 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:19.503496 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:19.503554 (+ 58us) server_negotiation.cc:407] Connection header received
0925 02:44:19.506795 (+ 3241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:19.506836 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:19.507578 (+ 742us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:19.508134 (+ 556us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:19.508762 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:19.510842 (+ 2080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:19.512372 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:19.513764 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:19.514019 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:19.515371 (+ 1352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:19.515383 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:19.515405 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:44:19.516188 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:19.516282 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:44:19.516310 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:19.516483 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:19.516740 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 02:44:19.517096 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":442,"threads_started":1}
I20260925 02:44:19.518637 10930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586: No bootstrap required, opened a new log
I20260925 02:44:19.522634 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:44:19.527431 10930 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d46fe45d1c4ef9ba63f222f89b9586" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43905 } }
I20260925 02:44:19.528250 10930 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:19.528404 10930 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2d46fe45d1c4ef9ba63f222f89b9586, State: Initialized, Role: FOLLOWER
I20260925 02:44:19.529152 10930 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d46fe45d1c4ef9ba63f222f89b9586" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43905 } }
I20260925 02:44:19.529435 10930 raft_consensus.cc:388] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:19.529629 10930 raft_consensus.cc:482] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:19.529811 10930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:19.532437 10930 raft_consensus.cc:504] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d46fe45d1c4ef9ba63f222f89b9586" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43905 } }
I20260925 02:44:19.533119 10930 leader_election.cc:302] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d2d46fe45d1c4ef9ba63f222f89b9586; no voters:
I20260925 02:44:19.533792 10930 leader_election.cc:288] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:19.533985 10935 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:19.534984 10935 raft_consensus.cc:686] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [term 1 LEADER]: Becoming Leader. State: Replica: d2d46fe45d1c4ef9ba63f222f89b9586, State: Running, Role: LEADER
I20260925 02:44:19.535631 10935 consensus_queue.cc:237] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d46fe45d1c4ef9ba63f222f89b9586" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43905 } }
I20260925 02:44:19.536332 10930 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:19.538942 10936 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d2d46fe45d1c4ef9ba63f222f89b9586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d46fe45d1c4ef9ba63f222f89b9586" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43905 } } }
I20260925 02:44:19.539362 10936 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:19.539023 10937 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d2d46fe45d1c4ef9ba63f222f89b9586. Latest consensus state: current_term: 1 leader_uuid: "d2d46fe45d1c4ef9ba63f222f89b9586" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d2d46fe45d1c4ef9ba63f222f89b9586" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43905 } } }
I20260925 02:44:19.539491 10937 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:19.542517 10942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:19.548738 10942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:19.559064 10942 catalog_manager.cc:1347] Generated new cluster ID: 6b2ba9d9b0c34bde8ef2b493b9a38f42
I20260925 02:44:19.559222 10942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:19.571929 10942 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:19.573604 10942 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:19.581833 10942 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d2d46fe45d1c4ef9ba63f222f89b9586: Generated new TSK 0
I20260925 02:44:19.583086 10942 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:44:19.666600 10933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:19.607445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58602 (local address 127.5.56.190:43905)
0925 02:44:19.607743 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:19.607753 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:19.608170 (+ 417us) server_negotiation.cc:407] Connection header received
0925 02:44:19.608460 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:19.608464 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:19.608541 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:19.608677 (+ 136us) server_negotiation.cc:226] Negotiated authn=JWT
0925 02:44:19.609325 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:19.610491 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:19.612701 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:19.613183 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:19.613293 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:19.613944 (+ 651us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 02:44:19.665845 (+ 51901us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:44:19.666259 (+ 414us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58602: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":90}
W20260925 02:44:19.666790 10933 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58602: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 02:44:19.667060 10959 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:19.607371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43905 (local address 127.0.0.1:58602)
0925 02:44:19.608047 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:19.608068 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:19.608241 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:19.608736 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:19.608743 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:19.608756 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0925 02:44:19.609102 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:19.609116 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:19.610704 (+ 1588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:19.610711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:19.612520 (+ 1809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:19.612540 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:19.613376 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:19.613382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:19.613583 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:19.613638 (+ 55us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 02:44:19.666198 (+ 52560us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 02:44:19.666618 (+ 420us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.5.56.190:43905: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":355,"threads_started":1}
I20260925 02:44:19.668277 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 10863
I20260925 02:44:19.669834 10925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:19.772684 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 10863
2026-09-25T02:44:19Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (812 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (440 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (446 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:44:20 dist-test-slave-n563 krb5kdc[10974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:44:20 dist-test-slave-n563 krb5kdc[10974](info): set up 2 sockets
Sep 25 02:44:20 dist-test-slave-n563 krb5kdc[10974](info): commencing operation
krb5kdc: starting...
W20260925 02:44:21.776845 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.005s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:44:21 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:44:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:44:22.011960 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:33357
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:33357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:22.367795 10987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:22.368203 10987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:22.368384 10987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:22.376215 10987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:22.376411 10987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:22.376520 10987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:22.376624 10987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:22.376711 10987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:22.387279 10987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:33357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:33357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:22.388844 10987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:22.390368 10987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:22.403944 10992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:22.404035 10993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:22.404040 10995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:22.404819 10987 server_base.cc:1034] running on GCE node
I20260925 02:44:22.405876 10987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:22.407665 10987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:22.408955 10987 hybrid_clock.cc:648] HybridClock initialized: now 1790304262408874 us; error 75 us; skew 500 ppm
I20260925 02:44:22.409389 10987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:22 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304262, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:22.413033 10987 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:44:22.414968 10987 webserver.cc:460] Webserver started at http://127.5.56.190:37049/ using document root <none> and password file <none>
I20260925 02:44:22.415814 10987 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:22.415990 10987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:22.416371 10987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:22.419804 10987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "d1b350fe46864ebc88a60fe9074a2104"
format_stamp: "Formatted at 2026-09-25 02:44:22 on dist-test-slave-n563"
server_key: "d6a6c0ef342c98684008a434f10b5c36"
server_key_iv: "388b668030ae319817f495abd2efc710"
server_key_version: "encryptionkey@0"
I20260925 02:44:22.420733 10987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "d1b350fe46864ebc88a60fe9074a2104"
format_stamp: "Formatted at 2026-09-25 02:44:22 on dist-test-slave-n563"
server_key: "d6a6c0ef342c98684008a434f10b5c36"
server_key_iv: "388b668030ae319817f495abd2efc710"
server_key_version: "encryptionkey@0"
I20260925 02:44:22.426997 10987 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260925 02:44:22.431586 11002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:22.433296 10987 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:44:22.433558 10987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "d1b350fe46864ebc88a60fe9074a2104"
format_stamp: "Formatted at 2026-09-25 02:44:22 on dist-test-slave-n563"
server_key: "d6a6c0ef342c98684008a434f10b5c36"
server_key_iv: "388b668030ae319817f495abd2efc710"
server_key_version: "encryptionkey@0"
I20260925 02:44:22.433816 10987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:22.456281 10987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:22.461390 10987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:22.461731 10987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:22.482223 10987 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:33357
I20260925 02:44:22.482254 11055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:33357 every 8 connection(s)
I20260925 02:44:22.483832 10987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:22.485532 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 10987
I20260925 02:44:22.485756 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:22.486500 5346 external_mini_cluster.cc:1428] Setting key fc8ceac51e06b2426a228e1edb21761c
I20260925 02:44:22.488675 11056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:44:22 dist-test-slave-n563 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304261, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:22.503978 11056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104: Bootstrap starting.
I20260925 02:44:22.508741 11056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:22.510097 11056 log.cc:824] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:22.514000 11056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104: No bootstrap required, opened a new log
I20260925 02:44:22.514132 11059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:22.488757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45754 (local address 127.5.56.190:33357)
0925 02:44:22.489684 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:22.489704 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:22.489764 (+ 60us) server_negotiation.cc:407] Connection header received
0925 02:44:22.492205 (+ 2441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:22.492241 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:22.492873 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:22.493417 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:22.494855 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:22.497067 (+ 2212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:22.498634 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:22.499761 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:22.499986 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:22.504566 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:22.504581 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:22.504605 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:22.504661 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:22.508885 (+ 4224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:22.509749 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:22.509759 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:22.509768 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:22.510028 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:22.510502 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:22.510507 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:22.510510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:22.511309 (+ 799us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:22.511561 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:22.512161 (+ 600us) server_negotiation.cc:299] Negotiation successful
0925 02:44:22.512645 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":358,"threads_started":1}
I20260925 02:44:22.522979 11056 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b350fe46864ebc88a60fe9074a2104" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33357 } }
I20260925 02:44:22.523442 11056 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:22.523599 11056 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b350fe46864ebc88a60fe9074a2104, State: Initialized, Role: FOLLOWER
I20260925 02:44:22.524346 11056 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b350fe46864ebc88a60fe9074a2104" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33357 } }
I20260925 02:44:22.524593 11056 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:22.524699 11056 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:22.524847 11056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:22.527251 11056 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b350fe46864ebc88a60fe9074a2104" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33357 } }
I20260925 02:44:22.527827 11056 leader_election.cc:302] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1b350fe46864ebc88a60fe9074a2104; no voters:
I20260925 02:44:22.528450 11056 leader_election.cc:288] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:22.528623 11061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:22.529378 11061 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [term 1 LEADER]: Becoming Leader. State: Replica: d1b350fe46864ebc88a60fe9074a2104, State: Running, Role: LEADER
I20260925 02:44:22.530016 11061 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b350fe46864ebc88a60fe9074a2104" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33357 } }
I20260925 02:44:22.530947 11056 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:22.532994 11063 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1b350fe46864ebc88a60fe9074a2104. Latest consensus state: current_term: 1 leader_uuid: "d1b350fe46864ebc88a60fe9074a2104" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b350fe46864ebc88a60fe9074a2104" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33357 } } }
I20260925 02:44:22.533406 11063 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:22.534915 11062 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1b350fe46864ebc88a60fe9074a2104" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b350fe46864ebc88a60fe9074a2104" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33357 } } }
I20260925 02:44:22.535341 11062 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:22.535620 11066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:22.541306 11066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:22.553390 11066 catalog_manager.cc:1347] Generated new cluster ID: 82a21bec04c947528a4765e2ab026b4d
I20260925 02:44:22.553599 11066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:22.562904 11066 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:22.564369 11066 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:22.580304 11066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1b350fe46864ebc88a60fe9074a2104: Generated new TSK 0
I20260925 02:44:22.581343 11066 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:44:22.650919 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33357
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:23.013777 11082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:23.014108 11082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:23.014214 11082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:23.022274 11082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:23.022496 11082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:23.022663 11082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:44:23.033432 11082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:23.034878 11082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:23.036295 11082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:23.051417 11087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:23.051419 11088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:23.051645 11090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:23.052068 11082 server_base.cc:1034] running on GCE node
I20260925 02:44:23.052847 11082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:23.053987 11082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:23.055233 11082 hybrid_clock.cc:648] HybridClock initialized: now 1790304263055187 us; error 50 us; skew 500 ppm
I20260925 02:44:23.055589 11082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:23 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304263, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:23.059149 11082 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:44:23.060884 11082 webserver.cc:460] Webserver started at http://127.5.56.129:43403/ using document root <none> and password file <none>
I20260925 02:44:23.061677 11082 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:23.061833 11082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:23.062194 11082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:23.065629 11082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "2c2cfeb84a584526adcff2121a03861b"
format_stamp: "Formatted at 2026-09-25 02:44:23 on dist-test-slave-n563"
server_key: "e94bbec6298ddeafd725895b7931bb5e"
server_key_iv: "af48ad1b8d553bcbc97df539f611e2dc"
server_key_version: "encryptionkey@0"
I20260925 02:44:23.066568 11082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "2c2cfeb84a584526adcff2121a03861b"
format_stamp: "Formatted at 2026-09-25 02:44:23 on dist-test-slave-n563"
server_key: "e94bbec6298ddeafd725895b7931bb5e"
server_key_iv: "af48ad1b8d553bcbc97df539f611e2dc"
server_key_version: "encryptionkey@0"
I20260925 02:44:23.072342 11082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:44:23.076395 11097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:23.077984 11082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:23.078204 11082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "2c2cfeb84a584526adcff2121a03861b"
format_stamp: "Formatted at 2026-09-25 02:44:23 on dist-test-slave-n563"
server_key: "e94bbec6298ddeafd725895b7931bb5e"
server_key_iv: "af48ad1b8d553bcbc97df539f611e2dc"
server_key_version: "encryptionkey@0"
I20260925 02:44:23.078485 11082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:23.104996 11082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:23.109511 11082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:23.109885 11082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:23.110994 11082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:23.112473 11082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:23.112581 11082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:23.112736 11082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:23.112810 11082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:23.148239 11082 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:40335
I20260925 02:44:23.148281 11211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:40335 every 8 connection(s)
I20260925 02:44:23.149677 11082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:44:23.156333 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 11082
I20260925 02:44:23.156558 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:44:23.157245 5346 external_mini_cluster.cc:1428] Setting key c36194ec03a7f485fd0fa371531b9174
Sep 25 02:44:23 dist-test-slave-n563 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304263, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:23.171643 11214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:23.152478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:42817 (local address 127.5.56.190:33357)
0925 02:44:23.152904 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:23.152909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:23.154016 (+ 1107us) server_negotiation.cc:407] Connection header received
0925 02:44:23.155997 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:23.156003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:23.156103 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:23.156232 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:23.158174 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:23.159053 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:23.160862 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:23.161289 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:23.161437 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:23.165247 (+ 3810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:23.165253 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:23.165256 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:23.165292 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:23.168158 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:23.169057 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:23.169062 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:23.169064 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:23.169192 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:23.169679 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:23.169687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:23.169690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:23.169975 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:23.170136 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:23.171006 (+ 870us) server_negotiation.cc:299] Negotiation successful
0925 02:44:23.171253 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":206,"threads_started":1}
I20260925 02:44:23.172786 11215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:23.153084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:33357 (local address 127.5.56.129:42817)
0925 02:44:23.153839 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:23.153885 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:23.155694 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:23.156537 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:23.156549 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:23.157166 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:23.157946 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:23.157986 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:23.159257 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:23.159262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:23.160697 (+ 1435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:23.160711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:23.161545 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:23.161554 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:23.161816 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:23.162588 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:23.162614 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:23.164960 (+ 2346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:23.168345 (+ 3385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:23.168357 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:23.168379 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:23.168886 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:23.169317 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:23.169322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:23.169326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:23.169545 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:23.170109 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:23.170118 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:23.170606 (+ 488us) client_negotiation.cc:769] Sending connection context
0925 02:44:23.170901 (+ 295us) client_negotiation.cc:240] Negotiation successful
0925 02:44:23.171516 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":316,"threads_started":1}
I20260925 02:44:23.174580 11212 heartbeater.cc:347] Connected to a master server at 127.5.56.190:33357
I20260925 02:44:23.174924 11212 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:23.175743 11212 heartbeater.cc:511] Master 127.5.56.190:33357 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
I20260925 02:44:23.178243 11019 ts_manager.cc:194] Registered new tserver with Master: 2c2cfeb84a584526adcff2121a03861b (127.5.56.129:40335)
I20260925 02:44:23.179874 11019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:42817
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:44:23.205832 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33357
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:23.558996 11218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:23.559320 11218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:23.559499 11218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:23.567237 11218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:23.567368 11218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:23.567584 11218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:44:23.577850 11218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:23.579311 11218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:23.580770 11218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:23.595522 11223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:23.595518 11226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:23.595553 11224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:23.596928 11218 server_base.cc:1034] running on GCE node
I20260925 02:44:23.597654 11218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:23.598783 11218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:23.600004 11218 hybrid_clock.cc:648] HybridClock initialized: now 1790304263599935 us; error 66 us; skew 500 ppm
I20260925 02:44:23.600373 11218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:23 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304263, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:23.603830 11218 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:44:23.605535 11218 webserver.cc:460] Webserver started at http://127.5.56.130:45355/ using document root <none> and password file <none>
I20260925 02:44:23.606318 11218 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:23.606513 11218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:23.606848 11218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:23.610208 11218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "dffbbec18e90467da303ee5cbf849587"
format_stamp: "Formatted at 2026-09-25 02:44:23 on dist-test-slave-n563"
server_key: "af6176194f992f1e6d2ce92b57b11f1d"
server_key_iv: "f12cbd3396e662823111764b4a08715e"
server_key_version: "encryptionkey@0"
I20260925 02:44:23.611143 11218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "dffbbec18e90467da303ee5cbf849587"
format_stamp: "Formatted at 2026-09-25 02:44:23 on dist-test-slave-n563"
server_key: "af6176194f992f1e6d2ce92b57b11f1d"
server_key_iv: "f12cbd3396e662823111764b4a08715e"
server_key_version: "encryptionkey@0"
I20260925 02:44:23.616688 11218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:44:23.620548 11233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:23.622066 11218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:23.622265 11218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "dffbbec18e90467da303ee5cbf849587"
format_stamp: "Formatted at 2026-09-25 02:44:23 on dist-test-slave-n563"
server_key: "af6176194f992f1e6d2ce92b57b11f1d"
server_key_iv: "f12cbd3396e662823111764b4a08715e"
server_key_version: "encryptionkey@0"
I20260925 02:44:23.622562 11218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:23.638898 11218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:23.643194 11218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:23.643558 11218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:23.644639 11218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:23.646188 11218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:23.646296 11218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:23.646483 11218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:23.646571 11218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:23.681530 11218 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:39883
I20260925 02:44:23.681578 11346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:39883 every 8 connection(s)
I20260925 02:44:23.683149 11218 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:44:23.689098 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 11218
I20260925 02:44:23.689308 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:44:23.689787 5346 external_mini_cluster.cc:1428] Setting key 854b5c3365b305344706c3017d9b3537
Sep 25 02:44:23 dist-test-slave-n563 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304263, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:23.704869 11349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:23.685826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:37563 (local address 127.5.56.190:33357)
0925 02:44:23.686349 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:23.686354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:23.687289 (+ 935us) server_negotiation.cc:407] Connection header received
0925 02:44:23.689201 (+ 1912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:23.689207 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:23.689312 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:23.689444 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:23.691336 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:23.692192 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:23.693815 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:23.694283 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:23.694458 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:23.698282 (+ 3824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:23.698291 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:23.698297 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:23.698351 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:23.701115 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:23.702187 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:23.702191 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:23.702193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:23.702329 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:23.702924 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:23.702927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:23.702929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:23.703321 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:23.703451 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:23.704402 (+ 951us) server_negotiation.cc:299] Negotiation successful
0925 02:44:23.704587 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":314,"threads_started":1}
I20260925 02:44:23.706588 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:23.686267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:33357 (local address 127.5.56.130:37563)
0925 02:44:23.687117 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:23.687156 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:23.688888 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:23.689719 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:23.689727 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:23.690340 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:23.691158 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:23.691178 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:23.692360 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:23.692364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:23.693656 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:23.693669 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:23.694561 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:23.694570 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:23.694822 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:23.695557 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:23.695583 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:23.697960 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:23.701297 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:23.701310 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:23.701334 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:23.702047 (+ 713us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:23.702499 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:23.702507 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:23.702513 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:23.702801 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:23.703457 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:23.703469 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:23.703996 (+ 527us) client_negotiation.cc:769] Sending connection context
0925 02:44:23.704328 (+ 332us) client_negotiation.cc:240] Negotiation successful
0925 02:44:23.704983 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":306,"threads_started":1}
I20260925 02:44:23.708335 11347 heartbeater.cc:347] Connected to a master server at 127.5.56.190:33357
I20260925 02:44:23.708666 11347 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:23.709553 11347 heartbeater.cc:511] Master 127.5.56.190:33357 requested a full tablet report, sending...
I20260925 02:44:23.711382 11019 ts_manager.cc:194] Registered new tserver with Master: dffbbec18e90467da303ee5cbf849587 (127.5.56.130:39883)
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
I20260925 02:44:23.712637 11019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:37563
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:44:23.743407 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33357
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36447
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:24.089427 11354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:24.090121 11354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:24.090345 11354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:24.103857 11354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:24.104110 11354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:24.104374 11354 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:44:24.122031 11354 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36447
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33357
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:24.123924 11354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:24.125695 11354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:24.143162 11359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:24.143195 11360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:24.144160 11362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:24.144313 11354 server_base.cc:1034] running on GCE node
I20260925 02:44:24.145008 11354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:24.146070 11354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:24.147300 11354 hybrid_clock.cc:648] HybridClock initialized: now 1790304264147226 us; error 76 us; skew 500 ppm
I20260925 02:44:24.147694 11354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:24.151486 11354 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:44:24.153486 11354 webserver.cc:460] Webserver started at http://127.5.56.131:43337/ using document root <none> and password file <none>
I20260925 02:44:24.154366 11354 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:24.154567 11354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:24.154891 11354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:24.158071 11354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "d89e2843ca554f0590102bbd780f07ef"
format_stamp: "Formatted at 2026-09-25 02:44:24 on dist-test-slave-n563"
server_key: "70b960e9eecc1ac70a35165f9f660ab0"
server_key_iv: "0830c22c8e2877dcc5aa7375a1d5716b"
server_key_version: "encryptionkey@0"
I20260925 02:44:24.159008 11354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "d89e2843ca554f0590102bbd780f07ef"
format_stamp: "Formatted at 2026-09-25 02:44:24 on dist-test-slave-n563"
server_key: "70b960e9eecc1ac70a35165f9f660ab0"
server_key_iv: "0830c22c8e2877dcc5aa7375a1d5716b"
server_key_version: "encryptionkey@0"
I20260925 02:44:24.164867 11354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:44:24.168852 11369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:24.170343 11354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:24.170533 11354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "d89e2843ca554f0590102bbd780f07ef"
format_stamp: "Formatted at 2026-09-25 02:44:24 on dist-test-slave-n563"
server_key: "70b960e9eecc1ac70a35165f9f660ab0"
server_key_iv: "0830c22c8e2877dcc5aa7375a1d5716b"
server_key_version: "encryptionkey@0"
I20260925 02:44:24.170728 11354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:24.183621 11212 heartbeater.cc:503] Master 127.5.56.190:33357 was elected leader, sending a full tablet report...
I20260925 02:44:24.190521 11354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:24.194757 11354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:24.195096 11354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:24.196123 11354 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:24.197603 11354 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:24.197708 11354 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:24.197866 11354 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:24.197940 11354 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:24.232002 11354 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:41547
I20260925 02:44:24.232039 11481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:41547 every 8 connection(s)
I20260925 02:44:24.233407 11354 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:44:24.236194 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 11354
I20260925 02:44:24.236400 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:44:24.237159 5346 external_mini_cluster.cc:1428] Setting key 5a934ac3c4e630ed201f3c75b54c209a
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:24.255309 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:24.236140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:50905 (local address 127.5.56.190:33357)
0925 02:44:24.238237 (+ 2097us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:24.238243 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:24.238262 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:44:24.240438 (+ 2176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:24.240447 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:24.240548 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:24.240631 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:24.242271 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.243113 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.244651 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.245053 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.245147 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:24.248682 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:24.248687 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:24.248690 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:24.248725 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:24.251915 (+ 3190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.252859 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.252866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.252869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.253034 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.253620 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.253625 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.253628 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.253933 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:24.254067 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:24.254841 (+ 774us) server_negotiation.cc:299] Negotiation successful
0925 02:44:24.255028 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1960,"thread_start_us":2221,"threads_started":1}
I20260925 02:44:24.256336 11485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:24.236719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:33357 (local address 127.5.56.131:50905)
0925 02:44:24.237820 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:24.237868 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:24.240099 (+ 2231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:24.240912 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:24.240921 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:24.241472 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:24.242080 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:24.242125 (+ 45us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.243247 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.243251 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:24.244510 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:24.244533 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.245247 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.245252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:24.245457 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:24.246100 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:24.246127 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:24.248444 (+ 2317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:24.252070 (+ 3626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:24.252082 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:24.252103 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:24.252730 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:24.253158 (+ 428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:24.253163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:24.253166 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:24.253478 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:24.254045 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:24.254052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:24.254479 (+ 427us) client_negotiation.cc:769] Sending connection context
0925 02:44:24.254746 (+ 267us) client_negotiation.cc:240] Negotiation successful
0925 02:44:24.255183 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":475,"threads_started":1}
I20260925 02:44:24.257809 11482 heartbeater.cc:347] Connected to a master server at 127.5.56.190:33357
I20260925 02:44:24.258133 11482 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:24.258987 11482 heartbeater.cc:511] Master 127.5.56.190:33357 requested a full tablet report, sending...
I20260925 02:44:24.260520 11019 ts_manager.cc:194] Registered new tserver with Master: d89e2843ca554f0590102bbd780f07ef (127.5.56.131:41547)
I20260925 02:44:24.261371 11019 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:50905
I20260925 02:44:24.265969 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/test-admin.keytab.
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:24.295735 5346 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 02:44:24.301522 11491 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:24.298761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35316 (local address 127.5.56.129:40335)
0925 02:44:24.299509 (+ 748us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:24.299516 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:24.299542 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:44:24.299604 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:24.299611 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:24.299837 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:24.300018 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:24.301029 (+ 1011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35316: BlockingRecv error: recv got EOF from 127.0.0.1:35316 (error 108)
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":349,"threads_started":1}
W20260925 02:44:24.301972 5346 init.cc:244] Error while opening credential cache '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5cc)
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:24.303716 5346 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:24.305222 5346 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:44:24.321528 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:24.307122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35330 (local address 127.5.56.129:40335)
0925 02:44:24.307339 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:24.307344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:24.307450 (+ 106us) server_negotiation.cc:407] Connection header received
0925 02:44:24.307643 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:24.307648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:24.307715 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:24.307809 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:24.308795 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.309808 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.311246 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.311874 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.311986 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:24.315753 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:24.315768 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:24.315785 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:24.315879 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:24.318795 (+ 2916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.319482 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.319489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.319492 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.319619 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.319938 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.319941 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.319943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.320326 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:24.320550 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:24.320896 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 02:44:24.321238 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 02:44:24.328208 11491 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:24.325806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.5.56.129:40335)
0925 02:44:24.325999 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:24.326004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:24.326431 (+ 427us) server_negotiation.cc:407] Connection header received
0925 02:44:24.326641 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:24.326646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:24.326771 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:24.326906 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:24.327939 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35340: BlockingRecv error: recv got EOF from 127.0.0.1:35340 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260925 02:44:24.328389 5346 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:24.330040 5346 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:24.331599 5346 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:44:24.346923 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:24.333157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.5.56.129:40335)
0925 02:44:24.333334 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:24.333338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:24.333613 (+ 275us) server_negotiation.cc:407] Connection header received
0925 02:44:24.333750 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:24.333754 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:24.333817 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:24.333907 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:24.335162 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.335916 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.337355 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.337787 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.337932 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:24.341692 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:24.341698 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:24.341701 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:24.341750 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:24.344372 (+ 2622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.345050 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.345059 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.345061 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.345187 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.345539 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.345542 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.345544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.345855 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:24.346066 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:24.346317 (+ 251us) server_negotiation.cc:299] Negotiation successful
0925 02:44:24.346585 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 02:44:24.351549 11491 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:24.349733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35354 (local address 127.5.56.129:40335)
0925 02:44:24.349966 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:24.349971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:24.350210 (+ 239us) server_negotiation.cc:407] Connection header received
0925 02:44:24.350334 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:24.350337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:24.350476 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:24.350554 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:24.351293 (+ 739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35354: BlockingRecv error: recv got EOF from 127.0.0.1:35354 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260925 02:44:24.351678 5346 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790304212743187-5346-0/krb5kdc/krb5cc' of type FILE
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:24.353319 5346 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:24.354806 5346 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:44:24.370596 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:24.356253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.5.56.129:40335)
0925 02:44:24.356501 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:24.356507 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:24.356692 (+ 185us) server_negotiation.cc:407] Connection header received
0925 02:44:24.356826 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:24.356830 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:24.356892 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:24.356981 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:24.357910 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.358719 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.359942 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.360346 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.360438 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:24.365411 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:24.365417 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:24.365420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:24.365460 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:24.368095 (+ 2635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.368802 (+ 707us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.368806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.368808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.368964 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.369291 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.369295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.369297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.369555 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:24.369787 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:24.369995 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 02:44:24.370188 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:44:24.372195 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 11082
I20260925 02:44:24.384248 11491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:24.372886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.5.56.129:40335)
0925 02:44:24.373152 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:24.373158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:24.373351 (+ 193us) server_negotiation.cc:407] Connection header received
0925 02:44:24.373462 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:24.373465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:24.373523 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:24.373599 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:24.374567 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.375320 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.376478 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.376914 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.377009 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:24.379224 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:24.379229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:24.379232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:24.379260 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:24.381603 (+ 2343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.382345 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.382348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.382350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.382554 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.382877 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.382880 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.382882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.383208 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:24.383406 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:24.383679 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 02:44:24.383854 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:44:24.384838 11207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:24.471685 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 11082
I20260925 02:44:24.489661 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 11218
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:44:24.506513 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:24.490604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48656 (local address 127.5.56.130:39883)
0925 02:44:24.491122 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:24.491130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:24.491155 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:44:24.491217 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:24.491223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:24.491443 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:24.491631 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:24.492689 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.493697 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.495149 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.495737 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.495908 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:24.500153 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:24.500168 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:24.500186 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:24.500228 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:24.503287 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.504076 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.504084 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.504086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.504227 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.504574 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.504578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.504580 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.504997 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:24.505370 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:24.505784 (+ 414us) server_negotiation.cc:299] Negotiation successful
0925 02:44:24.506115 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":262,"threads_started":1}
I20260925 02:44:24.507854 11342 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:24.593777 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 11218
I20260925 02:44:24.611639 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 11354
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[10974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304264, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:44:24.628829 11506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:24.612782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.5.56.131:41547)
0925 02:44:24.613436 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:24.613444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:24.613476 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:44:24.613541 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:24.613548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:24.613764 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:24.614012 (+ 248us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:24.615210 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.616292 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.617627 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:24.618245 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:24.618398 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:24.622772 (+ 4374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:24.622788 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:24.622804 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:24.622849 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:24.625759 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.626617 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.626625 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.626627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.626770 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:24.627211 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:24.627215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:24.627217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:24.627588 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:24.627920 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:24.628213 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 02:44:24.628500 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":332,"threads_started":1}
I20260925 02:44:24.630262 11477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:24.716528 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 11354
I20260925 02:44:24.734617 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 10987
I20260925 02:44:24.735877 11051 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:24.844980 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 10987
2026-09-25T02:44:24Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4191 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[11511](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[11511](info): set up 2 sockets
Sep 25 02:44:24 dist-test-slave-n563 krb5kdc[11511](info): commencing operation
krb5kdc: starting...
W20260925 02:44:25.952848 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.001s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:44:26 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:44:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:26Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.190.keytab.
Entry for principal oryx/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.190.keytab.
I20260925 02:44:26.191241 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:42051
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.190.keytab
--principal=oryx/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:42051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:26.548435 11524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:26.548769 11524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:26.548871 11524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:26.556488 11524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:26.556625 11524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:26.556672 11524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:26.556725 11524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:26.556769 11524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:26.567293 11524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:42051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.190.keytab
--principal=oryx/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:42051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:26.568753 11524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:26.570159 11524 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:26.582827 11529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:26.582793 11532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:26.582871 11530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:26.583878 11524 server_base.cc:1034] running on GCE node
I20260925 02:44:26.584820 11524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:26.586496 11524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:26.587759 11524 hybrid_clock.cc:648] HybridClock initialized: now 1790304266587694 us; error 55 us; skew 500 ppm
I20260925 02:44:26.588174 11524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:26 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304266, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:26.595587 11524 init.cc:377] Logged in from keytab as oryx/127.5.56.190@KRBTEST.COM (short username oryx)
I20260925 02:44:26.597400 11524 webserver.cc:460] Webserver started at http://127.5.56.190:46245/ using document root <none> and password file <none>
I20260925 02:44:26.598203 11524 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:26.598364 11524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:26.598740 11524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:26.601999 11524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "3d65122eeada4f58aa3c9176c0df7ba1"
format_stamp: "Formatted at 2026-09-25 02:44:26 on dist-test-slave-n563"
server_key: "e93b7707fd895f90cd8700aae0f8547a"
server_key_iv: "6f469ac45958d75cccd939b2ed64bed4"
server_key_version: "encryptionkey@0"
I20260925 02:44:26.602867 11524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "3d65122eeada4f58aa3c9176c0df7ba1"
format_stamp: "Formatted at 2026-09-25 02:44:26 on dist-test-slave-n563"
server_key: "e93b7707fd895f90cd8700aae0f8547a"
server_key_iv: "6f469ac45958d75cccd939b2ed64bed4"
server_key_version: "encryptionkey@0"
I20260925 02:44:26.608517 11524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:44:26.612528 11539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:26.614172 11524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:26.614390 11524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "3d65122eeada4f58aa3c9176c0df7ba1"
format_stamp: "Formatted at 2026-09-25 02:44:26 on dist-test-slave-n563"
server_key: "e93b7707fd895f90cd8700aae0f8547a"
server_key_iv: "6f469ac45958d75cccd939b2ed64bed4"
server_key_version: "encryptionkey@0"
I20260925 02:44:26.614671 11524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:26.636425 11524 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:26.641223 11524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:26.641557 11524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:26.661145 11524 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:42051
I20260925 02:44:26.661180 11590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:42051 every 8 connection(s)
I20260925 02:44:26.662711 11524 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:26.664601 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 11524
I20260925 02:44:26.664837 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:26.665521 5346 external_mini_cluster.cc:1428] Setting key c3115d2dd7a375bae7ad2a80cad27e50
I20260925 02:44:26.667307 11591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:26.679239 11591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1: Bootstrap starting.
Sep 25 02:44:26 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:26.682672 11591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:26.683794 11591 log.cc:824] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:26.686921 11591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1: No bootstrap required, opened a new log
I20260925 02:44:26.688300 11594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:26.667722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51930 (local address 127.5.56.190:42051)
0925 02:44:26.668533 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:26.668555 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:26.668599 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:44:26.670304 (+ 1705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:26.670330 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:26.670858 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:26.671310 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:26.672821 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:26.674173 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:26.675481 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:26.676237 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:26.676478 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:26.680588 (+ 4110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:26.680597 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:26.680616 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:26.680666 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:26.683922 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:26.684725 (+ 803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:26.684738 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:26.684744 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:26.684931 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:26.685344 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:26.685349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:26.685352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:26.685907 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:26.686232 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:26.686721 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 02:44:26.687210 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":363,"threads_started":1}
I20260925 02:44:26.696285 11591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } }
I20260925 02:44:26.696769 11591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:26.696900 11591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d65122eeada4f58aa3c9176c0df7ba1, State: Initialized, Role: FOLLOWER
I20260925 02:44:26.697544 11591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } }
I20260925 02:44:26.697834 11591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:26.697992 11591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:26.698168 11591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:26.700598 11591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } }
I20260925 02:44:26.701108 11591 leader_election.cc:302] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d65122eeada4f58aa3c9176c0df7ba1; no voters:
I20260925 02:44:26.701712 11591 leader_election.cc:288] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:26.701882 11596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:26.702593 11596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 1 LEADER]: Becoming Leader. State: Replica: 3d65122eeada4f58aa3c9176c0df7ba1, State: Running, Role: LEADER
I20260925 02:44:26.703204 11596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } }
I20260925 02:44:26.704030 11591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:26.706094 11597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } }
I20260925 02:44:26.707123 11597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:26.706583 11598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d65122eeada4f58aa3c9176c0df7ba1. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } }
I20260925 02:44:26.707436 11598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:26.712318 11602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:26.717136 11602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:26.726596 11602 catalog_manager.cc:1347] Generated new cluster ID: 714faf126fea4489854ae85ccb24415d
I20260925 02:44:26.726794 11602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:26.741243 11602 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:26.743335 11602 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:26.750978 11602 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1: Generated new TSK 0
I20260925 02:44:26.752185 11602 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab.
Entry for principal oryx/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab.
I20260925 02:44:26.813814 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42051
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab
--principal=oryx/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:27.167062 11617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:27.167377 11617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:27.167475 11617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:27.175015 11617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:27.175148 11617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:27.175277 11617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:44:27.185792 11617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab
--principal=oryx/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:27.187259 11617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:27.188823 11617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:27.203169 11622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:27.203096 11625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:27.203137 11623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:27.204499 11617 server_base.cc:1034] running on GCE node
I20260925 02:44:27.205320 11617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:27.206508 11617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:27.207751 11617 hybrid_clock.cc:648] HybridClock initialized: now 1790304267207696 us; error 50 us; skew 500 ppm
I20260925 02:44:27.208109 11617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:27 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304267, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:27.212013 11617 init.cc:377] Logged in from keytab as oryx/127.5.56.129@KRBTEST.COM (short username oryx)
I20260925 02:44:27.213795 11617 webserver.cc:460] Webserver started at http://127.5.56.129:37229/ using document root <none> and password file <none>
I20260925 02:44:27.214676 11617 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:27.214830 11617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:27.215170 11617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:27.218724 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "c088d4b02c704a7ba40c01da541a3fa8"
format_stamp: "Formatted at 2026-09-25 02:44:27 on dist-test-slave-n563"
server_key: "82f290711c5ee7046ea1e3bc441a26e8"
server_key_iv: "e06dc76993a56c5ab19f2aaed2c711a1"
server_key_version: "encryptionkey@0"
I20260925 02:44:27.219655 11617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "c088d4b02c704a7ba40c01da541a3fa8"
format_stamp: "Formatted at 2026-09-25 02:44:27 on dist-test-slave-n563"
server_key: "82f290711c5ee7046ea1e3bc441a26e8"
server_key_iv: "e06dc76993a56c5ab19f2aaed2c711a1"
server_key_version: "encryptionkey@0"
I20260925 02:44:27.225435 11617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:44:27.230734 11633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:27.233003 11617 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:44:27.233256 11617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "c088d4b02c704a7ba40c01da541a3fa8"
format_stamp: "Formatted at 2026-09-25 02:44:27 on dist-test-slave-n563"
server_key: "82f290711c5ee7046ea1e3bc441a26e8"
server_key_iv: "e06dc76993a56c5ab19f2aaed2c711a1"
server_key_version: "encryptionkey@0"
I20260925 02:44:27.233547 11617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:27.248878 11617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:27.253250 11617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:27.253528 11617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:27.254552 11617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:27.255939 11617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:27.256017 11617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:27.256103 11617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:27.256150 11617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:27.291318 11617 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:46809
I20260925 02:44:27.291365 11745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:46809 every 8 connection(s)
I20260925 02:44:27.292699 11617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:44:27.297945 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 11617
I20260925 02:44:27.298162 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:44:27.299017 5346 external_mini_cluster.cc:1428] Setting key a8d8ba5b3674cd2e448bc9966e300cc2
Sep 25 02:44:27 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304267, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:27.316596 11748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:27.295421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:59513 (local address 127.5.56.190:42051)
0925 02:44:27.295879 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:27.295884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:27.296911 (+ 1027us) server_negotiation.cc:407] Connection header received
0925 02:44:27.299540 (+ 2629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:27.299546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:27.299647 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:27.299780 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:27.301765 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:27.302562 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:27.305146 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:27.305567 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:27.305674 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:27.310492 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:27.310498 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:27.310502 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:27.310541 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:27.313290 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:27.314105 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:27.314111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:27.314116 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:27.314251 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:27.314737 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:27.314743 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:27.314746 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:27.315036 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:27.315199 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:27.315975 (+ 776us) server_negotiation.cc:299] Negotiation successful
0925 02:44:27.316214 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":216,"threads_started":1}
I20260925 02:44:27.317704 11749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:27.296027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.129:59513)
0925 02:44:27.296742 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:27.296780 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:27.299209 (+ 2429us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:27.300074 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:27.300086 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:27.300675 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:27.301573 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:27.301601 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:27.302726 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:27.302735 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:27.304987 (+ 2252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:27.305006 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:27.305831 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:27.305840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:27.306196 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:27.307036 (+ 840us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:27.307072 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:27.310217 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:27.313483 (+ 3266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:27.313494 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:27.313519 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:27.313971 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:27.314370 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:27.314374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:27.314378 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:27.314616 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:27.315170 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:27.315179 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:27.315577 (+ 398us) client_negotiation.cc:769] Sending connection context
0925 02:44:27.315852 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 02:44:27.316437 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":279,"threads_started":1}
I20260925 02:44:27.319583 11746 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42051
I20260925 02:44:27.320015 11746 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:27.321043 11746 heartbeater.cc:511] Master 127.5.56.190:42051 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.5.56.130@KRBTEST.COM; defaulting to no policy
I20260925 02:44:27.323737 11556 ts_manager.cc:194] Registered new tserver with Master: c088d4b02c704a7ba40c01da541a3fa8 (127.5.56.129:46809)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.130@KRBTEST.COM" created.
I20260925 02:44:27.325359 11556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:59513
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab.
Entry for principal oryx/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab.
I20260925 02:44:27.351660 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42051
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab
--principal=oryx/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:27.695647 11752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:27.696164 11752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:27.696339 11752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:27.704191 11752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:27.704317 11752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:27.704512 11752 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:44:27.715528 11752 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab
--principal=oryx/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:27.717025 11752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:27.718583 11752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:27.733174 11757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:27.733227 11758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:27.733435 11760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:27.734468 11752 server_base.cc:1034] running on GCE node
I20260925 02:44:27.735121 11752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:27.736159 11752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:27.737408 11752 hybrid_clock.cc:648] HybridClock initialized: now 1790304267737360 us; error 45 us; skew 500 ppm
I20260925 02:44:27.737769 11752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:27 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304267, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:27.741447 11752 init.cc:377] Logged in from keytab as oryx/127.5.56.130@KRBTEST.COM (short username oryx)
I20260925 02:44:27.743213 11752 webserver.cc:460] Webserver started at http://127.5.56.130:44141/ using document root <none> and password file <none>
I20260925 02:44:27.744001 11752 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:27.744113 11752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:27.744396 11752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:27.747653 11752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "feb3f982ef3a4a64a3ecf29fdea39345"
format_stamp: "Formatted at 2026-09-25 02:44:27 on dist-test-slave-n563"
server_key: "3347019c1fcb3fe4288543e884f6d81c"
server_key_iv: "5524d6843ab014235410508b4306d489"
server_key_version: "encryptionkey@0"
I20260925 02:44:27.748435 11752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "feb3f982ef3a4a64a3ecf29fdea39345"
format_stamp: "Formatted at 2026-09-25 02:44:27 on dist-test-slave-n563"
server_key: "3347019c1fcb3fe4288543e884f6d81c"
server_key_iv: "5524d6843ab014235410508b4306d489"
server_key_version: "encryptionkey@0"
I20260925 02:44:27.754251 11752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:44:27.758224 11767 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:27.759932 11752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:44:27.760095 11752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "feb3f982ef3a4a64a3ecf29fdea39345"
format_stamp: "Formatted at 2026-09-25 02:44:27 on dist-test-slave-n563"
server_key: "3347019c1fcb3fe4288543e884f6d81c"
server_key_iv: "5524d6843ab014235410508b4306d489"
server_key_version: "encryptionkey@0"
I20260925 02:44:27.760284 11752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:27.786763 11752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:27.791771 11752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:27.792048 11752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:27.793031 11752 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:27.794581 11752 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:27.794656 11752 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:27.794806 11752 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:27.794859 11752 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:27.831875 11752 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:35741
I20260925 02:44:27.831929 11880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:35741 every 8 connection(s)
I20260925 02:44:27.833288 11752 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:44:27.835243 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 11752
I20260925 02:44:27.835480 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:44:27.836128 5346 external_mini_cluster.cc:1428] Setting key 196d2bb635e115ce02af69c2aedcf236
Sep 25 02:44:27 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304267, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:27.855958 11883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:27.836437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:52919 (local address 127.5.56.190:42051)
0925 02:44:27.836972 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:27.836979 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:27.838470 (+ 1491us) server_negotiation.cc:407] Connection header received
0925 02:44:27.840404 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:27.840410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:27.840516 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:27.840638 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:27.842295 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:27.843139 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:27.844826 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:27.845287 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:27.845488 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:27.849676 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:27.849682 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:27.849685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:27.849728 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:27.852467 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:27.853346 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:27.853352 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:27.853359 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:27.853514 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:27.854003 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:27.854010 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:27.854013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:27.854303 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:27.854532 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:27.855304 (+ 772us) server_negotiation.cc:299] Negotiation successful
0925 02:44:27.855560 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":278,"threads_started":1}
WARNING: no policy specified for oryx/127.5.56.131@KRBTEST.COM; defaulting to no policy
I20260925 02:44:27.857142 11884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:27.837131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.130:52919)
0925 02:44:27.838029 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:27.838085 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:27.840110 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:27.840913 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:27.840923 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:27.841473 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:27.842101 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:27.842120 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:27.843328 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:27.843332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:27.844660 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:27.844674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:27.845592 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:27.845602 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:27.845870 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:27.846682 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:27.846707 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:27.849344 (+ 2637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:27.852657 (+ 3313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:27.852669 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:27.852690 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:27.853202 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:27.853648 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:27.853653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:27.853656 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:27.853875 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:27.854498 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:27.854507 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:27.854909 (+ 402us) client_negotiation.cc:769] Sending connection context
0925 02:44:27.855216 (+ 307us) client_negotiation.cc:240] Negotiation successful
0925 02:44:27.855824 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":386,"threads_started":1}
I20260925 02:44:27.858817 11881 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.131@KRBTEST.COM" created.
I20260925 02:44:27.859185 11881 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:27.860037 11881 heartbeater.cc:511] Master 127.5.56.190:42051 requested a full tablet report, sending...
I20260925 02:44:27.861724 11556 ts_manager.cc:194] Registered new tserver with Master: feb3f982ef3a4a64a3ecf29fdea39345 (127.5.56.130:35741)
I20260925 02:44:27.862640 11556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:52919
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab.
Entry for principal oryx/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab.
I20260925 02:44:27.886271 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42051
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab
--principal=oryx/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:28.238346 11887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:28.238848 11887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:28.239041 11887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:28.246685 11887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:28.246850 11887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:28.247040 11887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:44:28.257382 11887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab
--principal=oryx/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:28.258932 11887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:28.260385 11887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:28.275127 11895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:28.275362 11893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:28.275851 11892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:28.275959 11887 server_base.cc:1034] running on GCE node
I20260925 02:44:28.276763 11887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:28.277784 11887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:28.278995 11887 hybrid_clock.cc:648] HybridClock initialized: now 1790304268278908 us; error 66 us; skew 500 ppm
I20260925 02:44:28.279345 11887 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:28 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304268, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:28.282759 11887 init.cc:377] Logged in from keytab as oryx/127.5.56.131@KRBTEST.COM (short username oryx)
I20260925 02:44:28.284480 11887 webserver.cc:460] Webserver started at http://127.5.56.131:46465/ using document root <none> and password file <none>
I20260925 02:44:28.285269 11887 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:28.285422 11887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:28.285745 11887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:28.289062 11887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "d19a41448a394698b14bfc67ffd20931"
format_stamp: "Formatted at 2026-09-25 02:44:28 on dist-test-slave-n563"
server_key: "b1672712cc94d1b9a255f5a878729491"
server_key_iv: "f61401667f7b57b10f95c210ceb1d060"
server_key_version: "encryptionkey@0"
I20260925 02:44:28.289908 11887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "d19a41448a394698b14bfc67ffd20931"
format_stamp: "Formatted at 2026-09-25 02:44:28 on dist-test-slave-n563"
server_key: "b1672712cc94d1b9a255f5a878729491"
server_key_iv: "f61401667f7b57b10f95c210ceb1d060"
server_key_version: "encryptionkey@0"
I20260925 02:44:28.295723 11887 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:44:28.299584 11902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:28.301088 11887 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:44:28.301294 11887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "d19a41448a394698b14bfc67ffd20931"
format_stamp: "Formatted at 2026-09-25 02:44:28 on dist-test-slave-n563"
server_key: "b1672712cc94d1b9a255f5a878729491"
server_key_iv: "f61401667f7b57b10f95c210ceb1d060"
server_key_version: "encryptionkey@0"
I20260925 02:44:28.301527 11887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:28.315062 11887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:28.319918 11887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:28.320304 11887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:28.321404 11887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:28.323150 11887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:28.323266 11887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:28.323431 11887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:28.323498 11887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:28.328719 11746 heartbeater.cc:503] Master 127.5.56.190:42051 was elected leader, sending a full tablet report...
I20260925 02:44:28.361716 11887 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:34367
I20260925 02:44:28.361750 12014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:34367 every 8 connection(s)
I20260925 02:44:28.363262 11887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:44:28.369592 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 11887
I20260925 02:44:28.369782 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:44:28.370277 5346 external_mini_cluster.cc:1428] Setting key 9b4d0d38e6befb93887fdf825258bebb
Sep 25 02:44:28 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304268, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:28.386716 12017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:28.366306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:40319 (local address 127.5.56.190:42051)
0925 02:44:28.366779 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:28.366784 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:28.367921 (+ 1137us) server_negotiation.cc:407] Connection header received
0925 02:44:28.369893 (+ 1972us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:28.369899 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:28.370007 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:28.370149 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:28.372134 (+ 1985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.372958 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.374654 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.375078 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.375232 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:28.380057 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:28.380066 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:28.380071 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:28.380119 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:28.383160 (+ 3041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:28.383970 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:28.383975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:28.383979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:28.384141 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:28.384721 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:28.384733 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:28.384736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:28.385061 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:28.385240 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:28.386021 (+ 781us) server_negotiation.cc:299] Negotiation successful
0925 02:44:28.386300 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":222,"threads_started":1}
I20260925 02:44:28.387750 12018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:28.366954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.131:40319)
0925 02:44:28.367701 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:28.367750 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:28.369513 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:28.370482 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:28.370490 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:28.371222 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:28.371948 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:28.371970 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.373142 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.373146 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:28.374494 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:28.374507 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.375332 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.375340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:28.375611 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:28.376432 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:28.376457 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:28.379706 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:28.383316 (+ 3610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:28.383324 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:28.383341 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:28.383817 (+ 476us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:28.384281 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:28.384286 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:28.384289 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:28.384564 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:28.385205 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:28.385214 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:28.385623 (+ 409us) client_negotiation.cc:769] Sending connection context
0925 02:44:28.385912 (+ 289us) client_negotiation.cc:240] Negotiation successful
0925 02:44:28.386554 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":293,"threads_started":1}
I20260925 02:44:28.389252 12015 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42051
I20260925 02:44:28.389551 12015 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:28.390264 12015 heartbeater.cc:511] Master 127.5.56.190:42051 requested a full tablet report, sending...
I20260925 02:44:28.391845 11556 ts_manager.cc:194] Registered new tserver with Master: d19a41448a394698b14bfc67ffd20931 (127.5.56.131:34367)
I20260925 02:44:28.392630 11556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:40319
I20260925 02:44:28.398942 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.56.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.189.keytab.
Entry for principal oryx/127.5.56.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.189.keytab.
I20260925 02:44:28.452275 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.189:43763
--webserver_interface=127.5.56.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.189.keytab
--principal=oryx/127.5.56.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:42051,127.5.56.189:43763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:28.810019 12021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:28.810355 12021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:28.810518 12021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:28.818334 12021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:28.818560 12021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:28.818622 12021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:28.818681 12021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:28.818727 12021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:28.829213 12021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:42051,127.5.56.189:43763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.189.keytab
--principal=oryx/127.5.56.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.189:43763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.56.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
W20260925 02:44:28.829605 12021 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.5.56.190:42051,127.5.56.189:43763'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 02:44:28.830770 12021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:28.832584 12021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:28.846769 12026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:28.846719 12029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:28.846683 12027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:28.847909 12021 server_base.cc:1034] running on GCE node
I20260925 02:44:28.848877 12021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:28.850694 12021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:28.852003 12021 hybrid_clock.cc:648] HybridClock initialized: now 1790304268851913 us; error 36 us; skew 500 ppm
I20260925 02:44:28.852401 12021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:28 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304268, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:28.855937 12021 init.cc:377] Logged in from keytab as oryx/127.5.56.189@KRBTEST.COM (short username oryx)
I20260925 02:44:28.857678 12021 webserver.cc:460] Webserver started at http://127.5.56.189:36949/ using document root <none> and password file <none>
I20260925 02:44:28.858501 12021 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:28.858601 12021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:28.858881 12021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:28.862048 12021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data/instance:
uuid: "5701926ddb054d9cbc2012bf74a7c157"
format_stamp: "Formatted at 2026-09-25 02:44:28 on dist-test-slave-n563"
server_key: "4317ad07dc233576b95c9b1eead013ba"
server_key_iv: "bc2ab9b6a58953078a39e9081042913a"
server_key_version: "encryptionkey@0"
I20260925 02:44:28.862996 12021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/wal/instance:
uuid: "5701926ddb054d9cbc2012bf74a7c157"
format_stamp: "Formatted at 2026-09-25 02:44:28 on dist-test-slave-n563"
server_key: "4317ad07dc233576b95c9b1eead013ba"
server_key_iv: "bc2ab9b6a58953078a39e9081042913a"
server_key_version: "encryptionkey@0"
I20260925 02:44:28.865582 11881 heartbeater.cc:503] Master 127.5.56.190:42051 was elected leader, sending a full tablet report...
I20260925 02:44:28.869365 12021 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 02:44:28.873543 12036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:28.875267 12021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:44:28.875438 12021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/wal
uuid: "5701926ddb054d9cbc2012bf74a7c157"
format_stamp: "Formatted at 2026-09-25 02:44:28 on dist-test-slave-n563"
server_key: "4317ad07dc233576b95c9b1eead013ba"
server_key_iv: "bc2ab9b6a58953078a39e9081042913a"
server_key_version: "encryptionkey@0"
I20260925 02:44:28.875651 12021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:28.889134 12021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:28.897533 12021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:28.897909 12021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:28.925187 12089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.189:43763 every 8 connection(s)
I20260925 02:44:28.925220 12021 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.189:43763
I20260925 02:44:28.927728 12021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
I20260925 02:44:28.932276 12090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:28.936403 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 12021
I20260925 02:44:28.941363 12090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:44:28 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304268, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.189@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
Sep 25 02:44:28 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.56.189@KRBTEST.COM
I20260925 02:44:28.974436 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:28.947019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.5.56.190:42051)
0925 02:44:28.954934 (+ 7915us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:28.954941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:28.954966 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:44:28.955071 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:28.955076 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:28.955192 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:28.955334 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:28.957781 (+ 2447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.958664 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.960292 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.960910 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.961059 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:28.966461 (+ 5402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:28.966472 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:28.966476 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:28.966523 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:28.970083 (+ 3560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:28.971275 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:28.971284 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:28.971289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:28.971486 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:28.972136 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:28.972142 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:28.972145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:28.972619 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:28.972778 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:28.973921 (+ 1143us) server_negotiation.cc:299] Negotiation successful
0925 02:44:28.974123 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7779,"thread_start_us":7693,"threads_started":1}
I20260925 02:44:28.976293 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:28.946247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35310)
0925 02:44:28.947980 (+ 1733us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:28.948056 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:28.951515 (+ 3459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:28.955794 (+ 4279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:28.955814 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:28.956924 (+ 1110us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:28.957526 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:28.957559 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.958825 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.958830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:28.960112 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:28.960125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.961187 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.961197 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:28.961582 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:28.962524 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:28.962574 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:28.966155 (+ 3581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:28.970242 (+ 4087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:28.970260 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:28.970278 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:28.971106 (+ 828us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:28.971654 (+ 548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:28.971661 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:28.971667 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:28.971958 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:28.972791 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:28.972805 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:28.973419 (+ 614us) client_negotiation.cc:769] Sending connection context
0925 02:44:28.973792 (+ 373us) client_negotiation.cc:240] Negotiation successful
0925 02:44:28.974534 (+ 742us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1366,"thread_start_us":1169,"threads_started":1}
I20260925 02:44:28.977392 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:28.939083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.5.56.189:43763)
0925 02:44:28.954687 (+ 15604us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:28.954699 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:28.955542 (+ 843us) server_negotiation.cc:407] Connection header received
0925 02:44:28.955772 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:28.955779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:28.956031 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:28.956265 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:28.957941 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.960295 (+ 2354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.961833 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.962708 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.962863 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:28.968306 (+ 5443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:28.968317 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:28.968332 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:28.968367 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:28.973098 (+ 4731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:28.974732 (+ 1634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:28.974741 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:28.974744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:28.974893 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:28.975356 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:28.975363 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:28.975366 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:28.975709 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:28.976052 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:28.976370 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 02:44:28.976652 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15343,"thread_start_us":15069,"threads_started":1}
I20260925 02:44:28.978837 12090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:44:28 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304268, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.189@KRBTEST.COM for oryx/127.5.56.189@KRBTEST.COM
I20260925 02:44:28.994530 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:28.980154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.5.56.189:43763)
0925 02:44:28.980322 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:28.980329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:28.980350 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:44:28.980404 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:28.980408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:28.980493 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:28.980585 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:28.981415 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.982737 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.983917 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.984786 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.984949 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:28.988943 (+ 3994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:28.988948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:28.988951 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:28.988982 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:28.991623 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:28.992556 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:28.992560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:28.992562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:28.992675 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:28.993113 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:28.993116 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:28.993118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:28.993395 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:28.993508 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:28.994018 (+ 510us) server_negotiation.cc:299] Negotiation successful
0925 02:44:28.994221 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":54}
I20260925 02:44:28.995008 12094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:28.979664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:43763 (local address 127.0.0.1:38956)
0925 02:44:28.979901 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:28.979925 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:28.980046 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:28.980767 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:28.980773 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:28.981122 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:28.981255 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:28.981263 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.982888 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.982892 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:28.983780 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:28.983791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:28.984940 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:28.984944 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:28.985084 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:28.985742 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:28.985760 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:28.988688 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:28.991791 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:28.991801 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:28.991807 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:28.992427 (+ 620us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:28.992786 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:28.992794 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:28.992799 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:28.993002 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:28.993512 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:28.993520 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:28.993734 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 02:44:28.993927 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 02:44:28.994471 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57,"mutex_wait_us":102}
I20260925 02:44:28.999940 12090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: Bootstrap starting.
I20260925 02:44:29.003504 12090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:29.004745 12090 log.cc:824] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:29.007977 12090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: No bootstrap required, opened a new log
I20260925 02:44:29.015461 12090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } }
I20260925 02:44:29.015934 12090 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:29.016062 12090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5701926ddb054d9cbc2012bf74a7c157, State: Initialized, Role: FOLLOWER
I20260925 02:44:29.016705 12090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } }
I20260925 02:44:29.017693 12097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } } }
I20260925 02:44:29.018146 12097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:44:29.019140 12090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:44:29.029735 12108 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:44:29.029896 12108 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:44:29.033243 12021 master_runner.cc:428] Detected that this master 5701926ddb054d9cbc2012bf74a7c157 is joining an existing cluster
I20260925 02:44:29.033360 12021 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.56.189:43763
I20260925 02:44:29.068826 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.050757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38962 (local address 127.5.56.189:43763)
0925 02:44:29.050909 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.050914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.050934 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:44:29.051087 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.051093 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.051163 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.051245 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.053025 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.054531 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.057157 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.058099 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.058202 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.061025 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.061031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.061047 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.061082 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.065267 (+ 4185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.066223 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.066231 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.066234 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.066478 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.066915 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.066921 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.066924 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.067420 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.067581 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.068087 (+ 506us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.068350 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:44:29.069895 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.050054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:43763 (local address 127.0.0.1:38962)
0925 02:44:29.050734 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.050754 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.050902 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.051496 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.051501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.052344 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.052824 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.052844 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.054805 (+ 1961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.054813 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.056935 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.056956 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.058597 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.058603 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.058803 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.059600 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:29.059624 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:29.060829 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:29.065465 (+ 4636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.065469 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.065475 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.066090 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.066621 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.066626 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.066629 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.066790 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.067671 (+ 881us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:29.067678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:29.067839 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.069064 (+ 1225us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.069383 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":878,"threads_started":1}
I20260925 02:44:29.074846 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.050012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.5.56.190:42051)
0925 02:44:29.050611 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.050619 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.058497 (+ 7878us) server_negotiation.cc:407] Connection header received
0925 02:44:29.059073 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.059081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.059212 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.059273 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.060907 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.062019 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.064385 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.065109 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.065224 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.068006 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.068015 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.068019 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.068076 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.071985 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.072693 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.072699 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.072703 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.072849 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.073258 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.073263 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.073266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.073565 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.073739 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.074211 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.074483 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429}
I20260925 02:44:29.074930 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.051498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35320)
0925 02:44:29.058358 (+ 6860us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.058400 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.058865 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.059569 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.059573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.060328 (+ 755us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.060701 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.060713 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.062271 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.062276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.064182 (+ 1906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.064199 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.065568 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.065576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.065774 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.066523 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:29.066550 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:29.067763 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:29.072150 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.072157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.072161 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.072545 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.072967 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.072973 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.072977 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.073133 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.073690 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:29.073695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:29.073893 (+ 198us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.074078 (+ 185us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.074483 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6691,"thread_start_us":7028,"threads_started":1}
WARNING: no policy specified for oryx/127.5.56.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.188@KRBTEST.COM" created.
Sep 25 02:44:29 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304266, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.190@KRBTEST.COM for oryx/127.5.56.189@KRBTEST.COM
I20260925 02:44:29.099308 12093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.082450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.5.56.189:43763)
0925 02:44:29.082736 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.082745 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.083393 (+ 648us) server_negotiation.cc:407] Connection header received
0925 02:44:29.083793 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.083797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.083871 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.084001 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.085182 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.086051 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.087607 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.088183 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.088327 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.092020 (+ 3693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.092027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.092030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.092068 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.095218 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.096444 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.096450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.096454 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.096640 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.097140 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.097146 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.097149 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.097529 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.097697 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.098592 (+ 895us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.098862 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260925 02:44:29.100394 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.081945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:43763 (local address 127.0.0.1:38964)
0925 02:44:29.083258 (+ 1313us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.083296 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.083550 (+ 254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.084063 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.084071 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.084770 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.085011 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.085022 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.086203 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.086210 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.087447 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.087460 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.088352 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.088359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.088642 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.089425 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:29.089454 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:29.091786 (+ 2332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:29.095455 (+ 3669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.095478 (+ 23us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.095489 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.096269 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.096807 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.096814 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.096818 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.096999 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.097793 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:29.097803 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:29.098038 (+ 235us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.098273 (+ 235us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.099941 (+ 1668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":868,"threads_started":1}
I20260925 02:44:29.102954 11556 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.56.189:43763: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157"
member_type: NON_VOTER
last_known_addr {
host: "127.5.56.189"
port: 43763
}
attrs {
promote: true
}
}
dest_uuid: "3d65122eeada4f58aa3c9176c0df7ba1"
cas_config_opid_index: -1
I20260925 02:44:29.104563 11556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.188.keytab.
Entry for principal oryx/127.5.56.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.188.keytab.
I20260925 02:44:29.130716 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.188:41937
--webserver_interface=127.5.56.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.188.keytab
--principal=oryx/127.5.56.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:42051,127.5.56.189:43763,127.5.56.188:41937
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:44:29.133836 12120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5701926ddb054d9cbc2012bf74a7c157 (127.5.56.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } } }
I20260925 02:44:29.135514 12064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:29.139595 12064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Refusing update from remote peer 3d65122eeada4f58aa3c9176c0df7ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 02:44:29.151325 12124 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:44:29.154238 12043 tablet.cc:2369] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:44:29.161482 12120 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.56.189:43763
I20260925 02:44:29.163676 12021 master.cc:537] Master@127.5.56.189:43763 shutting down...
I20260925 02:44:29.164386 12121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } } }
I20260925 02:44:29.164748 12121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:29.224444 12064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 5701926ddb054d9cbc2012bf74a7c157 (127.5.56.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } } }
I20260925 02:44:29.232580 12097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d65122eeada4f58aa3c9176c0df7ba1. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } } }
I20260925 02:44:29.232877 12097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:44:29.233263 12128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } } }
I20260925 02:44:29.233556 12128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:44:29.235741 12131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } } }
I20260925 02:44:29.236003 12131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:44:29.236792 12021 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 LEARNER]: Raft consensus shutting down.
I20260925 02:44:29.237253 12021 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 LEARNER]: Raft consensus is shut down!
I20260925 02:44:29.237440 12021 tablet_replica.cc:330] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: stopping tablet replica
I20260925 02:44:29.244911 12021 master.cc:552] Master@127.5.56.189:43763 shutdown complete.
I20260925 02:44:29.245045 12021 master_runner.cc:305] Clearing existing system tablet
I20260925 02:44:29.245839 12021 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:29.245994 12021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:29.248463 12021 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:44:29.250015 12133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:29.250427 12021 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:29.250907 12021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260925 02:44:29.251017 12021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/wal
uuid: "5701926ddb054d9cbc2012bf74a7c157"
format_stamp: "Formatted at 2026-09-25 02:44:28 on dist-test-slave-n563"
server_key: "4317ad07dc233576b95c9b1eead013ba"
server_key_iv: "bc2ab9b6a58953078a39e9081042913a"
server_key_version: "encryptionkey@0"
I20260925 02:44:29.254477 12021 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:29.259370 12021 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:44:29.259584 12021 log.cc:1196] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 02:44:29.260061 12021 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: Deleting consensus metadata
I20260925 02:44:29.260331 12021 master_runner.cc:315] Copying system tablet from 127.5.56.190:42051
I20260925 02:44:29.263089 12021 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: tablet copy: Beginning tablet copy session from remote peer at address 127.5.56.190:42051
I20260925 02:44:29.277060 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.264204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35334 (local address 127.5.56.190:42051)
0925 02:44:29.264508 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.264517 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.265118 (+ 601us) server_negotiation.cc:407] Connection header received
0925 02:44:29.265331 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.265338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.265441 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.265558 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.266520 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.267290 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.268585 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.269055 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.269194 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.271356 (+ 2162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.271363 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.271368 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.271412 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.274125 (+ 2713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.274883 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.274889 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.274893 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.275038 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.275441 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.275446 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.275448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.275796 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.275962 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.276395 (+ 433us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.276640 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260925 02:44:29.277096 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.263996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35334)
0925 02:44:29.264974 (+ 978us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.264997 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.265196 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.265606 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.265613 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.266060 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.266293 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.266302 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.267447 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.267453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.268412 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.268424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.269233 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.269239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.269444 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.270125 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:29.270143 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:29.271155 (+ 1012us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:29.274274 (+ 3119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.274280 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.274284 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.274745 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.275160 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.275165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.275169 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.275312 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.275922 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:29.275928 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:29.276109 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.276257 (+ 148us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.276608 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":822,"thread_start_us":685,"threads_started":1}
I20260925 02:44:29.278316 11576 tablet_copy_service.cc:140] P 3d65122eeada4f58aa3c9176c0df7ba1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 5701926ddb054d9cbc2012bf74a7c157 ({username='oryx', principal='oryx/127.5.56.189@KRBTEST.COM'} at 127.0.0.1:35334)
I20260925 02:44:29.278769 11576 tablet_copy_service.cc:161] P 3d65122eeada4f58aa3c9176c0df7ba1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 5701926ddb054d9cbc2012bf74a7c157 at {username='oryx', principal='oryx/127.5.56.189@KRBTEST.COM'} at 127.0.0.1:35334: session id = 5701926ddb054d9cbc2012bf74a7c157-00000000000000000000000000000000
I20260925 02:44:29.281930 11576 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:44:29.284343 12021 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:44:29.289659 12021 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: tablet copy: Starting download of 0 data blocks...
I20260925 02:44:29.290019 12021 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: tablet copy: Starting download of 1 WAL segments...
I20260925 02:44:29.293588 12021 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:44:29.297948 11576 tablet_copy_service.cc:342] P 3d65122eeada4f58aa3c9176c0df7ba1: Request end of tablet copy session 5701926ddb054d9cbc2012bf74a7c157-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.56.189@KRBTEST.COM'} at 127.0.0.1:35334
I20260925 02:44:29.298183 11576 tablet_copy_service.cc:433] P 3d65122eeada4f58aa3c9176c0df7ba1: ending tablet copy session 5701926ddb054d9cbc2012bf74a7c157-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 5701926ddb054d9cbc2012bf74a7c157
I20260925 02:44:29.309887 12021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:29.312747 12141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:29.318359 12142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:29.319631 12144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:29.323820 12021 server_base.cc:1034] running on GCE node
I20260925 02:44:29.324242 12021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:29.330284 12021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:29.331570 12021 hybrid_clock.cc:648] HybridClock initialized: now 1790304269331505 us; error 51 us; skew 500 ppm
I20260925 02:44:29.331854 12021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:29 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304269, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:29.334156 12021 init.cc:377] Logged in from keytab as oryx/127.5.56.189@KRBTEST.COM (short username oryx)
I20260925 02:44:29.342748 12021 webserver.cc:460] Webserver started at http://127.5.56.189:38149/ using document root <none> and password file <none>
I20260925 02:44:29.343183 12021 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:29.343269 12021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:29.346565 12021 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260925 02:44:29.348768 12151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:29.349260 12021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:44:29.349411 12021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/wal
uuid: "5701926ddb054d9cbc2012bf74a7c157"
format_stamp: "Formatted at 2026-09-25 02:44:28 on dist-test-slave-n563"
server_key: "4317ad07dc233576b95c9b1eead013ba"
server_key_iv: "bc2ab9b6a58953078a39e9081042913a"
server_key_version: "encryptionkey@0"
I20260925 02:44:29.349586 12021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:29.376978 12021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:29.381352 12021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:29.395414 12015 heartbeater.cc:503] Master 127.5.56.190:42051 was elected leader, sending a full tablet report...
I20260925 02:44:29.429379 12021 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.189:43763
I20260925 02:44:29.429427 12202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.189:43763 every 8 connection(s)
I20260925 02:44:29.430289 12021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
I20260925 02:44:29.437671 12203 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:44:29.443511 12203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: Bootstrap starting.
I20260925 02:44:29.455199 12203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:44:29.455705 12203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: Bootstrap complete.
I20260925 02:44:29.456611 12203 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } }
I20260925 02:44:29.456843 12203 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5701926ddb054d9cbc2012bf74a7c157, State: Initialized, Role: LEARNER
I20260925 02:44:29.457091 12203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } }
I20260925 02:44:29.457918 12206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } } }
I20260925 02:44:29.458264 12203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:29.458693 12206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:44:29.467834 12217 catalog_manager.cc:1259] Loaded cluster ID: 714faf126fea4489854ae85ccb24415d
I20260925 02:44:29.467965 12217 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: loading cluster ID for follower catalog manager: success
I20260925 02:44:29.470717 12217 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: acquiring CA information for follower catalog manager: success
I20260925 02:44:29.472879 12217 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:44:29.614463 12122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:29.614845 12122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:29.615000 12122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:29.623081 12122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:29.623239 12122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:29.623327 12122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:29.623415 12122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:29.623517 12122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:29.634006 12122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:42051,127.5.56.189:43763,127.5.56.188:41937
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.188.keytab
--principal=oryx/127.5.56.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.188:41937
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.56.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:29.635728 12122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:29.637312 12122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:29.649837 12224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:29.650568 12223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:29.650599 12226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:29.651338 12122 server_base.cc:1034] running on GCE node
I20260925 02:44:29.652264 12122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:29.653877 12122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:29.655102 12122 hybrid_clock.cc:648] HybridClock initialized: now 1790304269655049 us; error 48 us; skew 500 ppm
I20260925 02:44:29.655474 12122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:29 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304269, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:29.658878 12122 init.cc:377] Logged in from keytab as oryx/127.5.56.188@KRBTEST.COM (short username oryx)
I20260925 02:44:29.660701 12122 webserver.cc:460] Webserver started at http://127.5.56.188:38309/ using document root <none> and password file <none>
I20260925 02:44:29.661612 12122 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:29.661760 12122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:29.662106 12122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:29.662978 12121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } } }
I20260925 02:44:29.663280 12121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:29.676208 12122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data/instance:
uuid: "fbfd3860e2ff46929abee5d1d5212edf"
format_stamp: "Formatted at 2026-09-25 02:44:29 on dist-test-slave-n563"
server_key: "cba06c1c4a6c1f983b8902b538820199"
server_key_iv: "5e7f618272f81d62ff509246d9d0f3f1"
server_key_version: "encryptionkey@0"
I20260925 02:44:29.677664 12122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/wal/instance:
uuid: "fbfd3860e2ff46929abee5d1d5212edf"
format_stamp: "Formatted at 2026-09-25 02:44:29 on dist-test-slave-n563"
server_key: "cba06c1c4a6c1f983b8902b538820199"
server_key_iv: "5e7f618272f81d62ff509246d9d0f3f1"
server_key_version: "encryptionkey@0"
I20260925 02:44:29.690136 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.663707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:43763 (local address 127.0.0.1:38978)
0925 02:44:29.678778 (+ 15071us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.678805 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.678981 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.683323 (+ 4342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.683328 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.683359 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:29.683549 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.683558 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.685863 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.685868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.687301 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.687318 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.689357 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.689367 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.689609 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.689656 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.689742 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.689855 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14930,"thread_start_us":14836,"threads_started":1}
I20260925 02:44:29.690294 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.663885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.5.56.189:43763)
0925 02:44:29.682923 (+ 19038us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.682927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.682953 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:44:29.683021 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.683025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.683117 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.683239 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:29.683749 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.685635 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.687488 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.689130 (+ 1642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.689253 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.689438 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.689798 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.689936 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18895,"thread_start_us":18747,"threads_started":1}
I20260925 02:44:29.690550 12122 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.010s sys 0.000s
I20260925 02:44:29.691285 12206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d65122eeada4f58aa3c9176c0df7ba1. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: NON_VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: true } } }
I20260925 02:44:29.691588 12206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: This master's current role is: LEARNER
I20260925 02:44:29.692358 12121 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1: attempting to promote NON_VOTER 5701926ddb054d9cbc2012bf74a7c157 to VOTER
I20260925 02:44:29.693372 12121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } }
I20260925 02:44:29.695240 12178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 LEARNER]: Refusing update from remote peer 3d65122eeada4f58aa3c9176c0df7ba1: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 02:44:29.695772 12235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:29.695873 12120 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 02:44:29.697777 12122 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260925 02:44:29.697983 12122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/wal
uuid: "fbfd3860e2ff46929abee5d1d5212edf"
format_stamp: "Formatted at 2026-09-25 02:44:29 on dist-test-slave-n563"
server_key: "cba06c1c4a6c1f983b8902b538820199"
server_key_iv: "5e7f618272f81d62ff509246d9d0f3f1"
server_key_version: "encryptionkey@0"
I20260925 02:44:29.698204 12122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:29.699421 12121 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5701926ddb054d9cbc2012bf74a7c157 (127.5.56.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } }
I20260925 02:44:29.700083 12178 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 5701926ddb054d9cbc2012bf74a7c157 (127.5.56.189) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } }
I20260925 02:44:29.703080 12206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } }
I20260925 02:44:29.703315 12206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:44:29.703521 12120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } }
I20260925 02:44:29.703789 12120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:29.704073 12120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } }
I20260925 02:44:29.704303 12120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:29.723467 12122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:29.727807 12122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:29.728121 12122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:29.748805 12122 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.188:41937
I20260925 02:44:29.749079 12287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.188:41937 every 8 connection(s)
I20260925 02:44:29.750370 12122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
I20260925 02:44:29.754522 12288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:29.756417 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 12122
I20260925 02:44:29.764412 12288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:44:29 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.56.188@KRBTEST.COM
Sep 25 02:44:29 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304269, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.188@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:29.782868 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.758911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52652 (local address 127.5.56.188:41937)
0925 02:44:29.759863 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.759879 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.759924 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:44:29.761709 (+ 1785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.761747 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.762232 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.762696 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.764284 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.765989 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.767838 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.768710 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.768891 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.773839 (+ 4948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.773847 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.773887 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.773943 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.777742 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.778592 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.778605 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.778617 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.778818 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.779266 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.779271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.779274 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.780081 (+ 807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.780411 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.780831 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.781319 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":352,"threads_started":1}
I20260925 02:44:29.784299 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.767415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35340 (local address 127.5.56.190:42051)
0925 02:44:29.767632 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.767639 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.768403 (+ 764us) server_negotiation.cc:407] Connection header received
0925 02:44:29.768600 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.768606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.768692 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.768808 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.770472 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.771348 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.773101 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.773809 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.773904 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.778077 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.778082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.778085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.778121 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.781421 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.782124 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.782130 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.782133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.782298 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.782713 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.782718 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.782721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.783140 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.783298 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.783737 (+ 439us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.783982 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 02:44:29.784353 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.767285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35340)
0925 02:44:29.768244 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.768287 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.768469 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.768895 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.768902 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.769885 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.770242 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.770257 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.771514 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.771521 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.772942 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.772962 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.774000 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.774006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.774236 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.774979 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:29.775006 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:29.777861 (+ 2855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:29.781551 (+ 3690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.781559 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.781565 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.782020 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.782467 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.782470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.782472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.782603 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.783279 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:29.783285 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:29.783437 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.783612 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.783932 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":520,"threads_started":1}
I20260925 02:44:29.785769 12288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:44:29 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304269, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.188@KRBTEST.COM for oryx/127.5.56.189@KRBTEST.COM
I20260925 02:44:29.798995 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.786586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:43763 (local address 127.0.0.1:38982)
0925 02:44:29.786785 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.786805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.786907 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.787282 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.787286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.787665 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.787798 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.787806 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.788831 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.788837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.789788 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.789799 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.790567 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.790573 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.790764 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.791310 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:29.791328 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:29.793480 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:29.796456 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.796464 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.796469 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.796861 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.797306 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.797311 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.797314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.797453 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.797978 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:29.797983 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:29.798128 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.798265 (+ 137us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.798634 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 02:44:29.798998 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.786703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.5.56.189:43763)
0925 02:44:29.786897 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.786903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.786923 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:44:29.787074 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.787080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.787158 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.787262 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.787946 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.788686 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.789931 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.790369 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.790548 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.793671 (+ 3123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.793684 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.793687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.793721 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.796320 (+ 2599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.797005 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.797011 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.797014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.797160 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.797572 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.797578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.797581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.797854 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.798007 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.798353 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.798635 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:44:29.800197 12288 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:44:29 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304269, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.188@KRBTEST.COM for oryx/127.5.56.188@KRBTEST.COM
I20260925 02:44:29.813024 12292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.800898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:41937 (local address 127.0.0.1:52656)
0925 02:44:29.801098 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.801112 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.801195 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.801473 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.801475 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.801800 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.801965 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.801973 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.803152 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.803155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.804042 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.804053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.804883 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.804887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.805103 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.805722 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:29.805740 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:29.807937 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:29.810960 (+ 3023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.810964 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.810967 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.811320 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.811631 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.811634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.811636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.811780 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.812251 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:29.812253 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:29.812369 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.812476 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.812676 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 02:44:29.813097 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.800902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52656 (local address 127.5.56.188:41937)
0925 02:44:29.801087 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.801097 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.801169 (+ 72us) server_negotiation.cc:407] Connection header received
0925 02:44:29.801276 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.801279 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.801346 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.801438 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.802093 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.803034 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.804172 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.804712 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.804903 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.808144 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.808149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.808153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.808182 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.810819 (+ 2637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.811416 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.811419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.811421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.811531 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.811878 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.811880 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.811882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.812154 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.812259 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.812564 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.812809 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":66}
I20260925 02:44:29.817837 12288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: Bootstrap starting.
I20260925 02:44:29.821249 12288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:29.822263 12288 log.cc:824] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:29.825021 12288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: No bootstrap required, opened a new log
I20260925 02:44:29.832253 12288 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } }
I20260925 02:44:29.832578 12288 raft_consensus.cc:374] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:29.832723 12288 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbfd3860e2ff46929abee5d1d5212edf, State: Initialized, Role: FOLLOWER
I20260925 02:44:29.833382 12288 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } }
I20260925 02:44:29.834272 12294 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } } }
I20260925 02:44:29.834681 12294 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:44:29.835418 12288 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:44:29.845800 12305 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:44:29.845981 12305 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:44:29.850512 12122 master_runner.cc:428] Detected that this master fbfd3860e2ff46929abee5d1d5212edf is joining an existing cluster
I20260925 02:44:29.850652 12122 master_runner.cc:432] Initiating AddMaster RPC to add 127.5.56.188:41937
I20260925 02:44:29.875429 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.855015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.5.56.190:42051)
0925 02:44:29.855200 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.855205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.855900 (+ 695us) server_negotiation.cc:407] Connection header received
0925 02:44:29.856224 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.856232 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.856331 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.856405 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.857932 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.859223 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.861784 (+ 2561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.862724 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.862854 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.867486 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.867495 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.867502 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.867562 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.871818 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.872625 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.872631 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.872635 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.872798 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.873270 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.873275 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.873279 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.873634 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.873753 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.874779 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.875044 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 02:44:29.877395 12311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.855044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35348)
0925 02:44:29.855785 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.855808 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.856050 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.856632 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.856637 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.857365 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.857751 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.857764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.859474 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.859481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.861582 (+ 2101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.861602 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.863281 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.863289 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.863578 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.864571 (+ 993us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:29.864603 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:29.866617 (+ 2014us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:29.871993 (+ 5376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.871998 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.872003 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.872483 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.872958 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.872963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.872967 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.873146 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.873854 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:29.873858 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:29.874104 (+ 246us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.876018 (+ 1914us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.876413 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":575,"thread_start_us":398,"threads_started":1}
I20260925 02:44:29.883428 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.856311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52658 (local address 127.5.56.188:41937)
0925 02:44:29.856554 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.856563 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.867842 (+ 11279us) server_negotiation.cc:407] Connection header received
0925 02:44:29.868044 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.868049 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.868111 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.868173 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.869495 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.870585 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.872809 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.873550 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.873639 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.876077 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.876085 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.876092 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.876136 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.880089 (+ 3953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.881011 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.881019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.881025 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.881209 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.881635 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.881641 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.881644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.882026 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.882196 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.882781 (+ 585us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.883060 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:44:29.884502 12312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.856691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:41937 (local address 127.0.0.1:52658)
0925 02:44:29.867748 (+ 11057us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.867771 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.867928 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.868365 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.868369 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.869022 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.869340 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.869354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.870818 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.870825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.872671 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.872686 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.873931 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.873937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.874109 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.874722 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:29.874751 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:29.875898 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:29.880249 (+ 4351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.880258 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.880262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.880805 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.881347 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.881352 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.881356 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.881501 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.882291 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:29.882298 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:29.882511 (+ 213us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.883675 (+ 1164us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.883981 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10895,"thread_start_us":10353,"threads_started":1}
I20260925 02:44:29.886188 12233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.855022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38984 (local address 127.5.56.189:43763)
0925 02:44:29.855214 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.855222 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.869972 (+ 14750us) server_negotiation.cc:407] Connection header received
0925 02:44:29.870212 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.870218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.870299 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.870363 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.871929 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.872994 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.875026 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.875733 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.875840 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.878518 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.878526 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.878531 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.878578 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.882832 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.883617 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.883622 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.883625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.883784 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.884231 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.884237 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.884240 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.884614 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.884796 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.885418 (+ 622us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.885673 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:44:29.887639 12313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.855588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:43763 (local address 127.0.0.1:38984)
0925 02:44:29.869856 (+ 14268us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.869876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.870074 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.870583 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.870588 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.871419 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.871728 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.871739 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.873166 (+ 1427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.873171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.874841 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.874855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.876134 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.876140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.876347 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.876973 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:29.876998 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:29.878276 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:29.882985 (+ 4709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.882990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.882995 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.883466 (+ 471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.883931 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.883936 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.883941 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.884097 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.884877 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:29.884881 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:29.885075 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.887074 (+ 1999us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.887310 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11704,"mutex_wait_us":71,"thread_start_us":11752,"threads_started":1}
I20260925 02:44:29.889526 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 11617
Sep 25 02:44:29 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304266, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.190@KRBTEST.COM for oryx/127.5.56.188@KRBTEST.COM
Sep 25 02:44:29 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.56.129@KRBTEST.COM
I20260925 02:44:29.903501 12291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.885903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52660 (local address 127.5.56.188:41937)
0925 02:44:29.886609 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.886616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.886633 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:44:29.886718 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.886722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.886811 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.886922 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.888145 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.889293 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.891562 (+ 2269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.892084 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.892888 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.896305 (+ 3417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.896313 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.896318 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.896364 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.900279 (+ 3915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.901241 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.901247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.901251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.901405 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.901890 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.901897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.901900 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.902242 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.902456 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.902853 (+ 397us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.903086 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549}
I20260925 02:44:29.903530 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.885595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:41937 (local address 127.0.0.1:52660)
0925 02:44:29.885979 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.885998 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.886114 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.886995 (+ 881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.887001 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.887507 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.887715 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.887749 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.889519 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.889526 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.890998 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.891022 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.892282 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.892290 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.892483 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.893245 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:29.893271 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:29.896097 (+ 2826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:29.900452 (+ 4355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.900458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.900463 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.901099 (+ 636us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.901532 (+ 433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:29.901537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:29.901541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:29.901767 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:29.902371 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:29.902376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:29.902596 (+ 220us) client_negotiation.cc:769] Sending connection context
0925 02:44:29.902752 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:44:29.903087 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20260925 02:44:29.905586 11556 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.5.56.188:41937: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf"
member_type: NON_VOTER
last_known_addr {
host: "127.5.56.188"
port: 41937
}
attrs {
promote: true
}
}
dest_uuid: "3d65122eeada4f58aa3c9176c0df7ba1"
cas_config_opid_index: 6
I20260925 02:44:29.906498 11556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true } }
I20260925 02:44:29.909070 12178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Refusing update from remote peer 3d65122eeada4f58aa3c9176c0df7ba1: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 02:44:29.909557 12263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:29.909811 12121 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 02:44:29.912513 12120 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fbfd3860e2ff46929abee5d1d5212edf (127.5.56.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true } } }
I20260925 02:44:29.912927 12314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.890704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36178 (local address 127.5.56.129:46809)
0925 02:44:29.891444 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.891454 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.891487 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:44:29.891558 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.891568 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.891862 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.892052 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.893617 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.894816 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.896686 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.897472 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.897632 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:29.902968 (+ 5336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:29.902994 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:29.903019 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:29.903069 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:29.907436 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.908466 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.908478 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.908481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.908655 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:29.909231 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:29.909238 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:29.909241 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:29.909759 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:29.910097 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:29.912188 (+ 2091us) server_negotiation.cc:299] Negotiation successful
0925 02:44:29.912476 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":362,"threads_started":1}
I20260925 02:44:29.913128 12178 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER fbfd3860e2ff46929abee5d1d5212edf (127.5.56.188) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true } } }
I20260925 02:44:29.914506 11741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:29.916544 12263 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 FOLLOWER]: Refusing update from remote peer 3d65122eeada4f58aa3c9176c0df7ba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 02:44:29.916779 12120 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.5.56.188:41937
I20260925 02:44:29.917090 12206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true } } }
I20260925 02:44:29.917383 12206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:44:29.917491 12122 master.cc:537] Master@127.5.56.188:41937 shutting down...
I20260925 02:44:29.918769 12124 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:29.919304 12121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true } } }
I20260925 02:44:29.919641 12121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:29.919461 12120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true } } }
I20260925 02:44:29.919874 12120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
W20260925 02:44:29.922601 11543 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 02:44:29.934710 11543 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 -> Peer fbfd3860e2ff46929abee5d1d5212edf (127.5.56.188:41937): Couldn't send request to peer fbfd3860e2ff46929abee5d1d5212edf. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 02:44:29.960855 12122 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:29.961372 12122 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:29.961591 12122 tablet_replica.cc:330] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: stopping tablet replica
I20260925 02:44:29.968905 12122 master.cc:552] Master@127.5.56.188:41937 shutdown complete.
I20260925 02:44:29.969090 12122 master_runner.cc:305] Clearing existing system tablet
I20260925 02:44:29.969579 12122 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:29.969784 12122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:29.972971 12122 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260925 02:44:29.974810 12319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:29.975201 12122 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:29.975665 12122 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260925 02:44:29.975819 12122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/wal
uuid: "fbfd3860e2ff46929abee5d1d5212edf"
format_stamp: "Formatted at 2026-09-25 02:44:29 on dist-test-slave-n563"
server_key: "cba06c1c4a6c1f983b8902b538820199"
server_key_iv: "5e7f618272f81d62ff509246d9d0f3f1"
server_key_version: "encryptionkey@0"
I20260925 02:44:29.981114 12122 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:29.988834 12122 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 02:44:29.989171 12122 log.cc:1196] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 02:44:29.989787 12122 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: Deleting consensus metadata
I20260925 02:44:29.990114 12122 master_runner.cc:315] Copying system tablet from 127.5.56.190:42051
I20260925 02:44:29.992974 12122 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: tablet copy: Beginning tablet copy session from remote peer at address 127.5.56.190:42051
I20260925 02:44:30.011214 12325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.993966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35362)
0925 02:44:29.994808 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:29.994829 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:29.995060 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:29.995628 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:29.995633 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:29.996214 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:29.996498 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:29.996516 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.998308 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:29.998314 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:29.999999 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:30.000014 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.001233 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.001239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:30.001427 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:30.002312 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:30.002337 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:30.003531 (+ 1194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:30.007981 (+ 4450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:30.007987 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:30.007992 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:30.008530 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:30.008979 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:30.008984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:30.008988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:30.009165 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:30.009918 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:30.009923 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:30.010120 (+ 197us) client_negotiation.cc:769] Sending connection context
0925 02:44:30.010263 (+ 143us) client_negotiation.cc:240] Negotiation successful
0925 02:44:30.010756 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":548,"threads_started":1}
I20260925 02:44:30.011228 12095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:29.994016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.5.56.190:42051)
0925 02:44:29.994291 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:29.994297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:29.994923 (+ 626us) server_negotiation.cc:407] Connection header received
0925 02:44:29.995218 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:29.995224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:29.995318 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:29.995388 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:29.996697 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:29.998096 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.000168 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.000844 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.000946 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:30.003768 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:30.003776 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:30.003782 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:30.003826 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:30.007817 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:30.008686 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:30.008693 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:30.008697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:30.008859 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:30.009288 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:30.009293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:30.009297 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:30.009786 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:30.009942 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:30.010431 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 02:44:30.010763 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260925 02:44:30.012202 11576 tablet_copy_service.cc:140] P 3d65122eeada4f58aa3c9176c0df7ba1: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fbfd3860e2ff46929abee5d1d5212edf ({username='oryx', principal='oryx/127.5.56.188@KRBTEST.COM'} at 127.0.0.1:35362)
I20260925 02:44:30.012390 11576 tablet_copy_service.cc:161] P 3d65122eeada4f58aa3c9176c0df7ba1: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fbfd3860e2ff46929abee5d1d5212edf at {username='oryx', principal='oryx/127.5.56.188@KRBTEST.COM'} at 127.0.0.1:35362: session id = fbfd3860e2ff46929abee5d1d5212edf-00000000000000000000000000000000
I20260925 02:44:30.016172 11576 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 02:44:30.018198 12122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 02:44:30.025514 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 11617
I20260925 02:44:30.025544 12122 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: tablet copy: Starting download of 0 data blocks...
I20260925 02:44:30.026412 12122 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: tablet copy: Starting download of 1 WAL segments...
I20260925 02:44:30.028923 12122 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 02:44:30.033730 11576 tablet_copy_service.cc:342] P 3d65122eeada4f58aa3c9176c0df7ba1: Request end of tablet copy session fbfd3860e2ff46929abee5d1d5212edf-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.5.56.188@KRBTEST.COM'} at 127.0.0.1:35362
I20260925 02:44:30.033905 11576 tablet_copy_service.cc:433] P 3d65122eeada4f58aa3c9176c0df7ba1: ending tablet copy session fbfd3860e2ff46929abee5d1d5212edf-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fbfd3860e2ff46929abee5d1d5212edf
I20260925 02:44:30.046451 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:46809
--local_ip_for_outbound_sockets=127.5.56.129
--tserver_master_addrs=127.5.56.190:42051
--webserver_port=37229
--webserver_interface=127.5.56.129
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab
--principal=oryx/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.56.190:42051,127.5.56.189:43763
--tserver_master_addrs=127.5.56.190:42051,127.5.56.189:43763,127.5.56.188:41937 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:44:30.051239 12122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:30.054602 12327 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:30.060127 12328 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:30.067625 12330 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:30.067974 12122 server_base.cc:1034] running on GCE node
I20260925 02:44:30.068569 12122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:30.069334 12122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:30.070523 12122 hybrid_clock.cc:648] HybridClock initialized: now 1790304270070469 us; error 43 us; skew 500 ppm
I20260925 02:44:30.070777 12122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:30 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304270, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:30.073232 12122 init.cc:377] Logged in from keytab as oryx/127.5.56.188@KRBTEST.COM (short username oryx)
I20260925 02:44:30.083591 12122 webserver.cc:460] Webserver started at http://127.5.56.188:36989/ using document root <none> and password file <none>
I20260925 02:44:30.083971 12122 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:30.084084 12122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:30.087539 12122 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260925 02:44:30.089648 12337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:30.090191 12122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260925 02:44:30.090319 12122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/wal
uuid: "fbfd3860e2ff46929abee5d1d5212edf"
format_stamp: "Formatted at 2026-09-25 02:44:29 on dist-test-slave-n563"
server_key: "cba06c1c4a6c1f983b8902b538820199"
server_key_iv: "5e7f618272f81d62ff509246d9d0f3f1"
server_key_version: "encryptionkey@0"
I20260925 02:44:30.090549 12122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:30.114846 12122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:30.119422 12122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:30.152575 12122 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.188:41937
I20260925 02:44:30.152643 12389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.188:41937 every 8 connection(s)
I20260925 02:44:30.153695 12122 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
I20260925 02:44:30.159133 12390 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:44:30.163424 12390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: Bootstrap starting.
I20260925 02:44:30.183883 12390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:44:30.184527 12390 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: Bootstrap complete.
I20260925 02:44:30.185819 12390 raft_consensus.cc:348] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true } }
I20260925 02:44:30.186213 12390 raft_consensus.cc:729] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fbfd3860e2ff46929abee5d1d5212edf, State: Initialized, Role: LEARNER
I20260925 02:44:30.186579 12390 consensus_queue.cc:260] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true } }
I20260925 02:44:30.187538 12393 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true } } }
I20260925 02:44:30.187918 12390 sys_catalog.cc:564] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:30.188048 12393 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: This master's current role is: LEARNER
I20260925 02:44:30.195618 12404 catalog_manager.cc:1259] Loaded cluster ID: 714faf126fea4489854ae85ccb24415d
I20260925 02:44:30.195850 12404 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: loading cluster ID for follower catalog manager: success
I20260925 02:44:30.199298 12404 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: acquiring CA information for follower catalog manager: success
I20260925 02:44:30.201714 12404 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:44:30.353900 12232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:30.344743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:41937 (local address 127.0.0.1:52666)
0925 02:44:30.345050 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:30.345070 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:30.345219 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:30.346320 (+ 1101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:30.346325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:30.346415 (+ 90us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:30.346649 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:30.346663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.348963 (+ 2300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.348969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:30.351029 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:30.351049 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.352990 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.352997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:30.353212 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:30.353252 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 02:44:30.353362 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:44:30.353537 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162}
I20260925 02:44:30.353981 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:30.344984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52666 (local address 127.5.56.188:41937)
0925 02:44:30.345858 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:30.345866 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:30.345890 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:44:30.345975 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:30.345982 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:30.346096 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:30.346243 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:30.346868 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.348754 (+ 1886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.351213 (+ 2459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.352767 (+ 1554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.352940 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:30.353242 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:30.353409 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 02:44:30.353578 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":507,"threads_started":1}
I20260925 02:44:30.356052 12365 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260925 02:44:30.356880 12393 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d65122eeada4f58aa3c9176c0df7ba1. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true } } }
I20260925 02:44:30.357287 12393 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: This master's current role is: LEARNER
W20260925 02:44:30.448199 12326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:30.448544 12326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:30.448745 12326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:30.456897 12326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:30.457024 12326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:30.457217 12326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:44:30.468295 12326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab
--principal=oryx/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:46809
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=37229
--tserver_master_addrs=127.5.56.190:42051,127.5.56.189:43763,127.5.56.188:41937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:30.469820 12326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:30.471382 12326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:30.486289 12411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:30.486408 12413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:30.486568 12410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:30.487267 12326 server_base.cc:1034] running on GCE node
I20260925 02:44:30.487979 12326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:30.489153 12326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:30.490438 12326 hybrid_clock.cc:648] HybridClock initialized: now 1790304270490328 us; error 46 us; skew 500 ppm
I20260925 02:44:30.490787 12326 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:30 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304270, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:30.494540 12326 init.cc:377] Logged in from keytab as oryx/127.5.56.129@KRBTEST.COM (short username oryx)
I20260925 02:44:30.496534 12326 webserver.cc:460] Webserver started at http://127.5.56.129:37229/ using document root <none> and password file <none>
I20260925 02:44:30.497341 12326 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:30.497468 12326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:30.503454 12326 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:44:30.506716 12420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:30.508286 12326 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:44:30.508502 12326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "c088d4b02c704a7ba40c01da541a3fa8"
format_stamp: "Formatted at 2026-09-25 02:44:27 on dist-test-slave-n563"
server_key: "82f290711c5ee7046ea1e3bc441a26e8"
server_key_iv: "e06dc76993a56c5ab19f2aaed2c711a1"
server_key_version: "encryptionkey@0"
I20260925 02:44:30.509635 12326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:30.524322 12326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:30.529337 12326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:30.529829 12326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:30.531008 12326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:30.532482 12326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:30.532608 12326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:30.532742 12326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:30.532831 12326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:30.569653 12326 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:46809
I20260925 02:44:30.569744 12533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:46809 every 8 connection(s)
I20260925 02:44:30.571280 12326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:44:30.574827 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 12326
I20260925 02:44:30.575364 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 11752
Sep 25 02:44:30 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.56.130@KRBTEST.COM
Sep 25 02:44:30 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304270, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.188@KRBTEST.COM
Sep 25 02:44:30 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304270, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.189@KRBTEST.COM
Sep 25 02:44:30 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304270, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:30.621577 12405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:30.588794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:48953 (local address 127.5.56.188:41937)
0925 02:44:30.589071 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:30.589080 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:30.598885 (+ 9805us) server_negotiation.cc:407] Connection header received
0925 02:44:30.599035 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:30.599042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:30.599138 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:30.599244 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:30.601676 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.602834 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.606147 (+ 3313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.606840 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.606949 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:30.613269 (+ 6320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:30.613278 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:30.613286 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:30.613338 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:30.617022 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:30.618158 (+ 1136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:30.618164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:30.618168 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:30.618334 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:30.619096 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:30.619101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:30.619105 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:30.619640 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:30.619765 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:30.620923 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0925 02:44:30.621213 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 02:44:30.622498 12539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:30.575296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:41019 (local address 127.5.56.190:42051)
0925 02:44:30.599040 (+ 23744us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:30.599049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:30.599072 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:44:30.599130 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:30.599135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:30.599213 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:30.599314 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:30.601682 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.602867 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.606158 (+ 3291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.606933 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.607068 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:30.613979 (+ 6911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:30.613989 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:30.613997 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:30.614054 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:30.618554 (+ 4500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:30.619687 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:30.619693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:30.619697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:30.619857 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:30.620340 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:30.620345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:30.620350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:30.620742 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:30.620892 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:30.621629 (+ 737us) server_negotiation.cc:299] Negotiation successful
0925 02:44:30.622126 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22724,"thread_start_us":22591,"threads_started":1}
I20260925 02:44:30.622627 12543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:30.577168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56436 (local address 127.5.56.130:35741)
0925 02:44:30.596679 (+ 19511us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:30.596695 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:30.596739 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:44:30.596838 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:30.596851 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:30.597290 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:30.597497 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:30.599972 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.601622 (+ 1650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.603615 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.604532 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.604728 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:30.612663 (+ 7935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:30.612684 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:30.612706 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:30.612764 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:30.617754 (+ 4990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:30.618858 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:30.618870 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:30.618873 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:30.619178 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:30.619711 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:30.619716 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:30.619718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:30.620191 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:30.620505 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:30.621757 (+ 1252us) server_negotiation.cc:299] Negotiation successful
0925 02:44:30.622051 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14688,"thread_start_us":13776,"threads_started":1}
I20260925 02:44:30.623064 12541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:30.575679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:57323 (local address 127.5.56.189:43763)
0925 02:44:30.583380 (+ 7701us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:30.583390 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:30.586412 (+ 3022us) server_negotiation.cc:407] Connection header received
0925 02:44:30.598263 (+ 11851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:30.598272 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:30.598420 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:30.598537 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:30.601684 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.603001 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.606217 (+ 3216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.606986 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.607115 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:30.614731 (+ 7616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:30.614741 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:30.614746 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:30.614794 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:30.618558 (+ 3764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:30.619612 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:30.619619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:30.619622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:30.619772 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:30.620300 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:30.620304 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:30.620307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:30.620668 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:30.620832 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:30.622475 (+ 1643us) server_negotiation.cc:299] Negotiation successful
0925 02:44:30.622721 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7479,"thread_start_us":7282,"threads_started":1}
I20260925 02:44:30.623498 12544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:30.588659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:41937 (local address 127.5.56.129:48953)
0925 02:44:30.598742 (+ 10083us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:30.598762 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:30.598898 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:30.599447 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:30.599453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:30.600395 (+ 942us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:30.601446 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:30.601475 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.603046 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.603051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:30.605969 (+ 2918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:30.605980 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.607314 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.607321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:30.607693 (+ 372us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:30.608687 (+ 994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:30.608729 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:30.612943 (+ 4214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:30.617230 (+ 4287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:30.617240 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:30.617264 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:30.617986 (+ 722us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:30.618556 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:30.618560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:30.618564 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:30.618940 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:30.619942 (+ 1002us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:30.619950 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:30.620456 (+ 506us) client_negotiation.cc:769] Sending connection context
0925 02:44:30.621850 (+ 1394us) client_negotiation.cc:240] Negotiation successful
0925 02:44:30.622109 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9933,"mutex_wait_us":395,"spinlock_wait_cycles":16000,"thread_start_us":9750,"threads_started":1}
I20260925 02:44:30.624277 11876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:30.624360 12538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:30.575030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:43763 (local address 127.5.56.129:57323)
0925 02:44:30.576775 (+ 1745us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:30.584445 (+ 7670us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:30.598080 (+ 13635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:30.599144 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:30.599156 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:30.600289 (+ 1133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:30.601453 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:30.601479 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.603212 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.603222 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:30.605965 (+ 2743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:30.605980 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.607426 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.607433 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:30.607697 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:30.608686 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:30.608728 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:30.614325 (+ 5597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:30.618716 (+ 4391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:30.618720 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:30.618723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:30.619465 (+ 742us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:30.619909 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:30.619915 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:30.619922 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:30.620163 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:30.620905 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:30.620909 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:30.621104 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 02:44:30.621279 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 02:44:30.621917 (+ 638us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1324,"mutex_wait_us":597,"spinlock_wait_cycles":26368,"thread_start_us":3972,"threads_started":1}
I20260925 02:44:30.624751 12540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:30.575030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.129:41019)
0925 02:44:30.588302 (+ 13272us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:30.588332 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:30.594920 (+ 6588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:30.599563 (+ 4643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:30.599568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:30.600377 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:30.601454 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:30.601479 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.603054 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.603061 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:30.605948 (+ 2887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:30.605966 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:30.607409 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:30.607417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:30.607691 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:30.608761 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:30.608788 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:30.613696 (+ 4908us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:30.618755 (+ 5059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:30.618762 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:30.618775 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:30.619518 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:30.619998 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:30.620002 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:30.620006 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:30.620213 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:30.620969 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:30.620974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:30.621130 (+ 156us) client_negotiation.cc:769] Sending connection context
0925 02:44:30.621299 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 02:44:30.623950 (+ 2651us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13009,"mutex_wait_us":2367,"spinlock_wait_cycles":137600,"thread_start_us":12405,"threads_started":1}
I20260925 02:44:30.625885 12534 heartbeater.cc:347] Connected to a master server at 127.5.56.188:41937
I20260925 02:44:30.626070 12536 heartbeater.cc:347] Connected to a master server at 127.5.56.189:43763
I20260925 02:44:30.626286 12536 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:30.626298 12534 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:30.627136 12535 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42051
I20260925 02:44:30.627313 12535 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:30.627357 12536 heartbeater.cc:511] Master 127.5.56.189:43763 requested a full tablet report, sending...
I20260925 02:44:30.627897 12535 heartbeater.cc:511] Master 127.5.56.190:42051 requested a full tablet report, sending...
I20260925 02:44:30.628691 12534 heartbeater.cc:511] Master 127.5.56.188:41937 requested a full tablet report, sending...
I20260925 02:44:30.629269 11556 ts_manager.cc:194] Re-registered known tserver with Master: c088d4b02c704a7ba40c01da541a3fa8 (127.5.56.129:46809)
I20260925 02:44:30.629889 12168 ts_manager.cc:194] Registered new tserver with Master: c088d4b02c704a7ba40c01da541a3fa8 (127.5.56.129:46809)
I20260925 02:44:30.630465 11556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:41019
I20260925 02:44:30.631166 12354 ts_manager.cc:194] Registered new tserver with Master: c088d4b02c704a7ba40c01da541a3fa8 (127.5.56.129:46809)
I20260925 02:44:30.721513 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 11752
I20260925 02:44:30.739321 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:35741
--local_ip_for_outbound_sockets=127.5.56.130
--tserver_master_addrs=127.5.56.190:42051
--webserver_port=44141
--webserver_interface=127.5.56.130
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab
--principal=oryx/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.56.190:42051,127.5.56.189:43763
--tserver_master_addrs=127.5.56.190:42051,127.5.56.189:43763,127.5.56.188:41937 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:44:30.758502 12124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: NON_VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: true } } }
I20260925 02:44:30.758965 12124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:30.760474 12548 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1: attempting to promote NON_VOTER fbfd3860e2ff46929abee5d1d5212edf to VOTER
I20260925 02:44:30.761111 12548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } }
I20260925 02:44:30.763361 12178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Refusing update from remote peer 3d65122eeada4f58aa3c9176c0df7ba1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 02:44:30.763988 12365 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 LEARNER]: Refusing update from remote peer 3d65122eeada4f58aa3c9176c0df7ba1: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260925 02:44:30.764148 12547 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 02:44:30.764679 12548 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260925 02:44:30.768733 12548 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fbfd3860e2ff46929abee5d1d5212edf (127.5.56.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:30.769410 12178 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fbfd3860e2ff46929abee5d1d5212edf (127.5.56.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:30.772239 12124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:30.772497 12124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:30.773047 12548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:30.773376 12548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:30.774135 12551 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:30.774573 12551 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:44:30.774870 12124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:30.775168 12124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:30.775537 12365 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, fbfd3860e2ff46929abee5d1d5212edf (127.5.56.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:30.778990 12393 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:30.779269 12393 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: This master's current role is: FOLLOWER
W20260925 02:44:31.100198 12546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:31.100544 12546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:31.100713 12546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:31.108700 12546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:31.108834 12546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:31.109025 12546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:44:31.120012 12546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab
--principal=oryx/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:35741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=44141
--tserver_master_addrs=127.5.56.190:42051,127.5.56.189:43763,127.5.56.188:41937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:31.121490 12546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:31.123216 12546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:31.138489 12560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:31.138489 12558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:31.138936 12557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:31.139434 12546 server_base.cc:1034] running on GCE node
I20260925 02:44:31.140100 12546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:31.141176 12546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:31.142467 12546 hybrid_clock.cc:648] HybridClock initialized: now 1790304271142349 us; error 48 us; skew 500 ppm
I20260925 02:44:31.142789 12546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:31 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304271, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:31.146224 12546 init.cc:377] Logged in from keytab as oryx/127.5.56.130@KRBTEST.COM (short username oryx)
I20260925 02:44:31.147894 12546 webserver.cc:460] Webserver started at http://127.5.56.130:44141/ using document root <none> and password file <none>
I20260925 02:44:31.148708 12546 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:31.148869 12546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:31.155267 12546 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:44:31.158510 12567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:31.160218 12546 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:44:31.160470 12546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "feb3f982ef3a4a64a3ecf29fdea39345"
format_stamp: "Formatted at 2026-09-25 02:44:27 on dist-test-slave-n563"
server_key: "3347019c1fcb3fe4288543e884f6d81c"
server_key_iv: "5524d6843ab014235410508b4306d489"
server_key_version: "encryptionkey@0"
I20260925 02:44:31.161696 12546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:31.185570 12546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:31.190824 12546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:31.191161 12546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:31.192355 12546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:31.193838 12546 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:31.193969 12546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:31.194110 12546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:31.194207 12546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:31.240216 12546 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:35741
I20260925 02:44:31.240288 12680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:35741 every 8 connection(s)
I20260925 02:44:31.242204 12546 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:44:31.245805 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 12546
I20260925 02:44:31.246106 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 11887
Sep 25 02:44:31 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304271, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.189@KRBTEST.COM
Sep 25 02:44:31 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304271, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.188@KRBTEST.COM
I20260925 02:44:31.298453 12689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.248157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:59319 (local address 127.5.56.189:43763)
0925 02:44:31.272144 (+ 23987us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:31.272151 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:31.272897 (+ 746us) server_negotiation.cc:407] Connection header received
0925 02:44:31.274584 (+ 1687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:31.274589 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:31.274690 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:31.274795 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:31.278062 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.279589 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.282464 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.283287 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.283420 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.289879 (+ 6459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:31.289887 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:31.289893 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:31.289946 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:31.293758 (+ 3812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.295343 (+ 1585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.295349 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.295352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.295561 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.296142 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.296149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.296154 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.296589 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:31.296734 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:31.297750 (+ 1016us) server_negotiation.cc:299] Negotiation successful
0925 02:44:31.298010 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23846,"thread_start_us":23996,"threads_started":1}
I20260925 02:44:31.298602 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.247155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:53165 (local address 127.5.56.188:41937)
0925 02:44:31.268887 (+ 21732us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:31.268897 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:31.273605 (+ 4708us) server_negotiation.cc:407] Connection header received
0925 02:44:31.273684 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:31.273689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:31.273772 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:31.273875 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:31.279770 (+ 5895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.280976 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.284522 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.285240 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.285343 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.290782 (+ 5439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:31.290789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:31.290794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:31.290840 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:31.293899 (+ 3059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.295137 (+ 1238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.295144 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.295147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.295344 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.295982 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.295989 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.295994 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.296411 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:31.296588 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:31.297748 (+ 1160us) server_negotiation.cc:299] Negotiation successful
0925 02:44:31.298063 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21523,"thread_start_us":21758,"threads_started":1}
Sep 25 02:44:31 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304266, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.56.131@KRBTEST.COM
I20260925 02:44:31.301347 12684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.246428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:41937 (local address 127.5.56.130:53165)
0925 02:44:31.269827 (+ 23399us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:31.269901 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:31.273093 (+ 3192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:31.277744 (+ 4651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:31.277752 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:31.278264 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:31.278561 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.278574 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.281219 (+ 2645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.281225 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.282698 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.282711 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.285875 (+ 3164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.285884 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.286121 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.286792 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:31.286825 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:31.290469 (+ 3644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:31.294067 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.294076 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.294081 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.294971 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.295504 (+ 533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.295509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.295513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.295816 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.296699 (+ 883us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:31.296709 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:31.297254 (+ 545us) client_negotiation.cc:769] Sending connection context
0925 02:44:31.298932 (+ 1678us) client_negotiation.cc:240] Negotiation successful
0925 02:44:31.299602 (+ 670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22914,"mutex_wait_us":2,"spinlock_wait_cycles":8192,"thread_start_us":28936,"threads_started":1}
I20260925 02:44:31.301725 12687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.247602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:43763 (local address 127.5.56.130:59319)
0925 02:44:31.272684 (+ 25082us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:31.272712 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:31.274421 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:31.275399 (+ 978us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:31.275416 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:31.276631 (+ 1215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:31.277784 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.277813 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.279845 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.279852 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.282236 (+ 2384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.282257 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.283901 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.283913 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.284329 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.285639 (+ 1310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:31.285669 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:31.289530 (+ 3861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:31.293954 (+ 4424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.293967 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.293990 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.295124 (+ 1134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.295731 (+ 607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.295736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.295740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.295975 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.296817 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:31.296822 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:31.297243 (+ 421us) client_negotiation.cc:769] Sending connection context
0925 02:44:31.300453 (+ 3210us) client_negotiation.cc:240] Negotiation successful
0925 02:44:31.301121 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24950,"mutex_wait_us":6,"spinlock_wait_cycles":43648,"thread_start_us":25744,"threads_started":1}
I20260925 02:44:31.303826 12683 heartbeater.cc:347] Connected to a master server at 127.5.56.189:43763
I20260925 02:44:31.304040 12681 heartbeater.cc:347] Connected to a master server at 127.5.56.188:41937
I20260925 02:44:31.304188 12683 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:31.304191 12681 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:31.305197 12683 heartbeater.cc:511] Master 127.5.56.189:43763 requested a full tablet report, sending...
I20260925 02:44:31.306236 12681 heartbeater.cc:511] Master 127.5.56.188:41937 requested a full tablet report, sending...
I20260925 02:44:31.307519 12354 ts_manager.cc:194] Registered new tserver with Master: feb3f982ef3a4a64a3ecf29fdea39345 (127.5.56.130:35741)
I20260925 02:44:31.307547 12168 ts_manager.cc:194] Registered new tserver with Master: feb3f982ef3a4a64a3ecf29fdea39345 (127.5.56.130:35741)
Sep 25 02:44:31 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304271, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:31.311440 12688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.247740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41714 (local address 127.5.56.131:34367)
0925 02:44:31.274412 (+ 26672us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:31.274421 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:31.274791 (+ 370us) server_negotiation.cc:407] Connection header received
0925 02:44:31.275175 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:31.275185 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:31.275467 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:31.275638 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:31.281686 (+ 6048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.283830 (+ 2144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.286547 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.289248 (+ 2701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.289407 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.300426 (+ 11019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:31.300449 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:31.300468 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:31.300523 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:31.308129 (+ 7606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.309040 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.309051 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.309055 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.309212 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.309643 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.309649 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.309652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.310065 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:31.310416 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:31.310757 (+ 341us) server_negotiation.cc:299] Negotiation successful
0925 02:44:31.311049 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25492,"thread_start_us":25319,"threads_started":1}
I20260925 02:44:31.312813 12010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:31.316970 12692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.276172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:56665 (local address 127.5.56.190:42051)
0925 02:44:31.288165 (+ 11993us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:31.288172 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:31.288199 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:44:31.288265 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:31.288271 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:31.288362 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:31.288486 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:31.290286 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.291610 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.299121 (+ 7511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.299931 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.300071 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.309008 (+ 8937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:31.309014 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:31.309017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:31.309057 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:31.312122 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.314533 (+ 2411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.314539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.314543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.314806 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.315337 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.315346 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.315351 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.315846 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:31.316023 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:31.316457 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 02:44:31.316709 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11844,"thread_start_us":11722,"threads_started":1}
I20260925 02:44:31.317186 12691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.276048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.130:56665)
0925 02:44:31.285229 (+ 9181us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:31.285250 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:31.285371 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:31.288779 (+ 3408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:31.288785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:31.289499 (+ 714us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:31.289979 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.289993 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.294663 (+ 4670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.294670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.298448 (+ 3778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.298468 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.300151 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.300158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.300400 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.301110 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:31.301133 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:31.308762 (+ 7629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:31.312275 (+ 3513us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.312281 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.312285 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.312850 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.314974 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.314981 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.314986 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.315189 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.315983 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:31.315991 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:31.316205 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 02:44:31.316360 (+ 155us) client_negotiation.cc:240] Negotiation successful
0925 02:44:31.316732 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9028,"thread_start_us":9434,"threads_started":1}
I20260925 02:44:31.318150 12682 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42051
I20260925 02:44:31.318318 12682 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:31.318853 12682 heartbeater.cc:511] Master 127.5.56.190:42051 requested a full tablet report, sending...
I20260925 02:44:31.319972 11556 ts_manager.cc:194] Re-registered known tserver with Master: feb3f982ef3a4a64a3ecf29fdea39345 (127.5.56.130:35741)
I20260925 02:44:31.320926 11556 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:56665
I20260925 02:44:31.402480 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 11887
W20260925 02:44:31.420313 11541 connection.cc:535] server connection from 127.5.56.131:40319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:44:31.421028 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:34367
--local_ip_for_outbound_sockets=127.5.56.131
--tserver_master_addrs=127.5.56.190:42051
--webserver_port=46465
--webserver_interface=127.5.56.131
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab
--principal=oryx/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.5.56.190:42051,127.5.56.189:43763
--tserver_master_addrs=127.5.56.190:42051,127.5.56.189:43763,127.5.56.188:41937 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:44:31.634632 12535 heartbeater.cc:503] Master 127.5.56.190:42051 was elected leader, sending a full tablet report...
W20260925 02:44:31.767323 12694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:31.767632 12694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:31.767819 12694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:31.775887 12694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:31.776032 12694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:31.776221 12694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:44:31.786643 12694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab
--principal=oryx/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:34367
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=46465
--tserver_master_addrs=127.5.56.190:42051,127.5.56.189:43763,127.5.56.188:41937
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:31.788607 12694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:31.790119 12694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:31.804404 12699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:31.804426 12702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:31.804453 12700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:31.805899 12694 server_base.cc:1034] running on GCE node
I20260925 02:44:31.806689 12694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:31.807706 12694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:31.808950 12694 hybrid_clock.cc:648] HybridClock initialized: now 1790304271808896 us; error 47 us; skew 500 ppm
I20260925 02:44:31.809303 12694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:31 dist-test-slave-n563 krb5kdc[11511](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304271, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:31.816993 12694 init.cc:377] Logged in from keytab as oryx/127.5.56.131@KRBTEST.COM (short username oryx)
I20260925 02:44:31.818931 12694 webserver.cc:460] Webserver started at http://127.5.56.131:46465/ using document root <none> and password file <none>
I20260925 02:44:31.819787 12694 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:31.819979 12694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:31.825735 12694 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260925 02:44:31.828672 12709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:31.830104 12694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:31.830292 12694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "d19a41448a394698b14bfc67ffd20931"
format_stamp: "Formatted at 2026-09-25 02:44:28 on dist-test-slave-n563"
server_key: "b1672712cc94d1b9a255f5a878729491"
server_key_iv: "f61401667f7b57b10f95c210ceb1d060"
server_key_version: "encryptionkey@0"
I20260925 02:44:31.831384 12694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:31.848701 12694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:31.853248 12694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:31.853540 12694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:31.854651 12694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:31.856104 12694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:31.856235 12694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:31.856360 12694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:31.856448 12694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:31.891067 12694 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:34367
I20260925 02:44:31.891220 12821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:34367 every 8 connection(s)
I20260925 02:44:31.892701 12694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:44:31.893780 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 12694
Sep 25 02:44:31 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304271, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.188@KRBTEST.COM
Sep 25 02:44:31 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304271, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.189@KRBTEST.COM
Sep 25 02:44:31 dist-test-slave-n563 krb5kdc[11511](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304271, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:31.944808 12829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.896675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:41937 (local address 127.5.56.131:50453)
0925 02:44:31.903400 (+ 6725us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:31.903457 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:31.914961 (+ 11504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:31.916736 (+ 1775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:31.916749 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:31.917752 (+ 1003us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:31.918931 (+ 1179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.918966 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.921071 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.921079 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.923625 (+ 2546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.923664 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.925163 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.925170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.925527 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.927750 (+ 2223us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:31.927788 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:31.933346 (+ 5558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:31.938554 (+ 5208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.938564 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.938585 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.939202 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.939702 (+ 500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.939707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.939711 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.940004 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.940730 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:31.940739 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:31.942184 (+ 1445us) client_negotiation.cc:769] Sending connection context
0925 02:44:31.942537 (+ 353us) client_negotiation.cc:240] Negotiation successful
0925 02:44:31.943162 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6409,"thread_start_us":6174,"threads_started":1}
I20260925 02:44:31.946112 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.897400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:50453 (local address 127.5.56.188:41937)
0925 02:44:31.902808 (+ 5408us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:31.902814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:31.904002 (+ 1188us) server_negotiation.cc:407] Connection header received
0925 02:44:31.915704 (+ 11702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:31.915712 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:31.915801 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:31.915921 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:31.919191 (+ 3270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.920715 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.923849 (+ 3134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.924610 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.924714 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.933629 (+ 8915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:31.933639 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:31.933647 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:31.933694 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:31.938365 (+ 4671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.939370 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.939379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.939382 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.939564 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.940143 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.940148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.940151 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.940502 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:31.940651 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:31.945390 (+ 4739us) server_negotiation.cc:299] Negotiation successful
0925 02:44:31.945668 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5269,"thread_start_us":5180,"threads_started":1}
I20260925 02:44:31.947172 12838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.910509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35368 (local address 127.5.56.190:42051)
0925 02:44:31.915770 (+ 5261us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:31.915776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:31.915791 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:44:31.915838 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:31.915843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:31.915943 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:31.918420 (+ 2477us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:31.918689 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.920205 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.924337 (+ 4132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.925269 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.930843 (+ 5574us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.934568 (+ 3725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:31.934576 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:31.934583 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:31.934627 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:31.938895 (+ 4268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.940338 (+ 1443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.940345 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.940350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.940551 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.941805 (+ 1254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.941811 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.941816 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.942222 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:31.942584 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:31.946544 (+ 3960us) server_negotiation.cc:299] Negotiation successful
0925 02:44:31.946802 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5124,"thread_start_us":4993,"threads_started":1}
I20260925 02:44:31.947489 12822 heartbeater.cc:347] Connected to a master server at 127.5.56.188:41937
I20260925 02:44:31.947870 12822 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:31.948865 12822 heartbeater.cc:511] Master 127.5.56.188:41937 requested a full tablet report, sending...
I20260925 02:44:31.950321 12837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.904245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:32929 (local address 127.5.56.190:42051)
0925 02:44:31.909096 (+ 4851us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:31.909104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:31.914261 (+ 5157us) server_negotiation.cc:407] Connection header received
0925 02:44:31.916668 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:31.916676 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:31.916757 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:31.916836 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:31.926174 (+ 9338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.927460 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.931971 (+ 4511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.932870 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.932988 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.941077 (+ 8089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:31.941086 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:31.941092 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:31.941135 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:31.944982 (+ 3847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.947453 (+ 2471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.947460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.947464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.947605 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.948053 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.948058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.948060 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.948480 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:31.948624 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:31.949599 (+ 975us) server_negotiation.cc:299] Negotiation successful
0925 02:44:31.949832 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4732,"thread_start_us":4630,"threads_started":1}
I20260925 02:44:31.950886 12354 ts_manager.cc:194] Registered new tserver with Master: d19a41448a394698b14bfc67ffd20931 (127.5.56.131:34367)
I20260925 02:44:31.952430 12830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.903827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38988 (local address 127.5.56.189:43763)
0925 02:44:31.914764 (+ 10937us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:31.914772 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:31.914791 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:44:31.914861 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:31.914867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:31.914947 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:31.916865 (+ 1918us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:31.926044 (+ 9179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.927607 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.930022 (+ 2415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.930923 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.931042 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.940866 (+ 9824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:31.940875 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:31.940882 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:31.940928 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:31.947662 (+ 6734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.948899 (+ 1237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.948907 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.948913 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.949081 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.950190 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.950195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.950198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.950556 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:31.950815 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:31.951475 (+ 660us) server_negotiation.cc:299] Negotiation successful
0925 02:44:31.951710 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10795,"thread_start_us":7107,"threads_started":1}
I20260925 02:44:31.953246 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.897070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:39079 (local address 127.5.56.189:43763)
0925 02:44:31.911193 (+ 14123us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:31.911201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:31.911228 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:44:31.922828 (+ 11600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:31.922834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:31.922932 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:31.923006 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:31.926444 (+ 3438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.927711 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.932914 (+ 5203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.933600 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.933685 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.941146 (+ 7461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:31.941155 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:31.941160 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:31.941206 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:31.945205 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.947793 (+ 2588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.947797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.947800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.947959 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.949191 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.949195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.949198 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.949519 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:31.949722 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:31.952618 (+ 2896us) server_negotiation.cc:299] Negotiation successful
0925 02:44:31.952858 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13958,"thread_start_us":17428,"threads_started":1}
I20260925 02:44:31.953857 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.904135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.131:32929)
0925 02:44:31.914145 (+ 10010us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:31.914169 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:31.914961 (+ 792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:31.922068 (+ 7107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:31.922078 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:31.924835 (+ 2757us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:31.925178 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.925188 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.927840 (+ 2652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.927845 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.931835 (+ 3990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.931850 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.934831 (+ 2981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.934837 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.935026 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.938293 (+ 3267us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:31.938315 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:31.940877 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:31.945543 (+ 4666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.945548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.945551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.946116 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.947745 (+ 1629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.947752 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.947758 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.947917 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.949023 (+ 1106us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:31.949027 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:31.949177 (+ 150us) client_negotiation.cc:769] Sending connection context
0925 02:44:31.951027 (+ 1850us) client_negotiation.cc:240] Negotiation successful
0925 02:44:31.951369 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9862,"thread_start_us":10148,"threads_started":1}
I20260925 02:44:31.953955 12831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.897011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:43763 (local address 127.5.56.131:39079)
0925 02:44:31.909259 (+ 12248us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:31.909285 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:31.922586 (+ 13301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:31.923285 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:31.923317 (+ 32us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:31.923961 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:31.924217 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.924229 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.928849 (+ 4620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.928854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.930187 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:31.930200 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.934016 (+ 3816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.934022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:31.934211 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.935923 (+ 1712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:31.935951 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:31.940206 (+ 4255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:31.946624 (+ 6418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.946628 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.946632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.947288 (+ 656us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.948687 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:31.948695 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:31.948701 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:31.948876 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:31.950775 (+ 1899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:31.950779 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:31.950930 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 02:44:31.952683 (+ 1753us) client_negotiation.cc:240] Negotiation successful
0925 02:44:31.952977 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12054,"thread_start_us":12436,"threads_started":1}
I20260925 02:44:31.955585 12824 heartbeater.cc:347] Connected to a master server at 127.5.56.189:43763
I20260925 02:44:31.955652 12823 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42051
I20260925 02:44:31.955744 12823 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:31.955753 12824 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:31.956313 12824 heartbeater.cc:511] Master 127.5.56.189:43763 requested a full tablet report, sending...
I20260925 02:44:31.956357 12823 heartbeater.cc:511] Master 127.5.56.190:42051 requested a full tablet report, sending...
I20260925 02:44:31.957531 12168 ts_manager.cc:194] Registered new tserver with Master: d19a41448a394698b14bfc67ffd20931 (127.5.56.131:34367)
I20260925 02:44:31.958004 11555 ts_manager.cc:194] Re-registered known tserver with Master: d19a41448a394698b14bfc67ffd20931 (127.5.56.131:34367)
I20260925 02:44:31.959542 11555 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:32929
I20260925 02:44:31.963310 12842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.915623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.5.56.188:41937)
0925 02:44:31.938036 (+ 22413us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:31.938044 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:31.938063 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:44:31.939127 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:31.939133 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:31.939240 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:31.939336 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:31.945203 (+ 5867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.948126 (+ 2923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.950438 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:31.951116 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:31.951244 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:31.956405 (+ 5161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:31.956415 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:31.956422 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:31.956467 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:31.960216 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.961048 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.961053 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.961057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.961271 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:31.961725 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:31.961730 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:31.961733 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:31.962071 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:31.962354 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:31.962801 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 02:44:31.962988 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22240,"thread_start_us":22446,"threads_started":1}
I20260925 02:44:31.956979 11556 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35368:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:44:31.966780 11556 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:44:32.023823 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.988327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36190 (local address 127.5.56.129:46809)
0925 02:44:32.011695 (+ 23368us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:32.011706 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:32.014808 (+ 3102us) server_negotiation.cc:407] Connection header received
0925 02:44:32.014980 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:32.014988 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:32.015232 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:32.015445 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:32.015982 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.016959 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.020956 (+ 3997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.022191 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.022353 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.022571 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:32.023265 (+ 694us) server_negotiation.cc:299] Negotiation successful
0925 02:44:32.023466 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23059,"thread_start_us":23371,"threads_started":1}
I20260925 02:44:32.023995 12853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.987222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:35741 (local address 127.0.0.1:56440)
0925 02:44:32.013280 (+ 26058us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:32.013300 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:32.013477 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:32.014881 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:32.014902 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:32.014940 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:32.015429 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:32.015446 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.017908 (+ 2462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.017913 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:32.020396 (+ 2483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:32.020413 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.022839 (+ 2426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.022847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:32.023041 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.023103 (+ 62us) client_negotiation.cc:769] Sending connection context
0925 02:44:32.023217 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:44:32.023474 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25885,"mutex_wait_us":94,"thread_start_us":27312,"threads_started":1}
I20260925 02:44:32.024369 12854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.987808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46809 (local address 127.0.0.1:36190)
0925 02:44:32.014656 (+ 26848us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:32.014674 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:32.014837 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:32.015437 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:32.015441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:32.015467 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:32.015809 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:32.015818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.017518 (+ 1700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.017522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:32.020809 (+ 3287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:32.020819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.022803 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.022811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:32.023026 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.023085 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 02:44:32.023191 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:44:32.023484 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26727,"mutex_wait_us":127,"thread_start_us":25142,"threads_started":1}
I20260925 02:44:32.024631 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.987625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56440 (local address 127.5.56.130:35741)
0925 02:44:32.012429 (+ 24804us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:32.012442 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:32.013649 (+ 1207us) server_negotiation.cc:407] Connection header received
0925 02:44:32.013810 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:32.013823 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:32.014284 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:32.014571 (+ 287us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:32.015690 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.017730 (+ 2040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.020574 (+ 2844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.022071 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.022196 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.022469 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:32.024133 (+ 1664us) server_negotiation.cc:299] Negotiation successful
0925 02:44:32.024317 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24552,"thread_start_us":24817,"threads_started":1}
I20260925 02:44:32.028151 12468 tablet_service.cc:1467] Processing CreateTablet for tablet 235ee5c20c3c4625ab62d53fe5d82740 (DEFAULT_TABLE table=test-table [id=fbfe091b7733437d94c3163e52368f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:32.028258 12615 tablet_service.cc:1467] Processing CreateTablet for tablet 235ee5c20c3c4625ab62d53fe5d82740 (DEFAULT_TABLE table=test-table [id=fbfe091b7733437d94c3163e52368f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:32.029436 12468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235ee5c20c3c4625ab62d53fe5d82740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:32.029496 12615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235ee5c20c3c4625ab62d53fe5d82740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:32.035959 12857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.986846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:34367 (local address 127.0.0.1:41718)
0925 02:44:32.012586 (+ 25740us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:32.012615 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:32.012827 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:32.019287 (+ 6460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:32.019292 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:32.019319 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:32.019861 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:32.019881 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.022539 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.022543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:32.025114 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:32.025128 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.034947 (+ 9819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.034956 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:32.035179 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.035222 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 02:44:32.035343 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 02:44:32.035523 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25521,"mutex_wait_us":30,"thread_start_us":31073,"threads_started":1}
I20260925 02:44:32.035977 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:31.987427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41718 (local address 127.5.56.131:34367)
0925 02:44:32.018676 (+ 31249us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:32.018682 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:32.018708 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:44:32.018826 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:32.018833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:32.019077 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:32.019307 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:32.020182 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.021763 (+ 1581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.025283 (+ 3520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.027241 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.027371 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.027587 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:32.035421 (+ 7834us) server_negotiation.cc:299] Negotiation successful
0925 02:44:32.035624 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31078,"thread_start_us":30896,"threads_started":1}
I20260925 02:44:32.040243 12757 tablet_service.cc:1467] Processing CreateTablet for tablet 235ee5c20c3c4625ab62d53fe5d82740 (DEFAULT_TABLE table=test-table [id=fbfe091b7733437d94c3163e52368f55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:32.041880 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235ee5c20c3c4625ab62d53fe5d82740. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:32.042446 12859 tablet_bootstrap.cc:492] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: Bootstrap starting.
I20260925 02:44:32.045841 12859 tablet_bootstrap.cc:654] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:32.047245 12860 tablet_bootstrap.cc:492] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: Bootstrap starting.
I20260925 02:44:32.047313 12859 log.cc:824] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:32.053251 12860 tablet_bootstrap.cc:654] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:32.055025 12860 log.cc:824] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:32.057415 12859 tablet_bootstrap.cc:492] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: No bootstrap required, opened a new log
I20260925 02:44:32.057819 12859 ts_tablet_manager.cc:1397] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260925 02:44:32.058351 12863 tablet_bootstrap.cc:492] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: Bootstrap starting.
I20260925 02:44:32.059664 12860 tablet_bootstrap.cc:492] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: No bootstrap required, opened a new log
I20260925 02:44:32.059957 12860 ts_tablet_manager.cc:1397] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20260925 02:44:32.062501 12863 tablet_bootstrap.cc:654] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:32.063882 12863 log.cc:824] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:32.068128 12863 tablet_bootstrap.cc:492] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: No bootstrap required, opened a new log
I20260925 02:44:32.068447 12863 ts_tablet_manager.cc:1397] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260925 02:44:32.068854 12860 raft_consensus.cc:348] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:32.069378 12860 raft_consensus.cc:374] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:32.069485 12860 raft_consensus.cc:729] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c088d4b02c704a7ba40c01da541a3fa8, State: Initialized, Role: FOLLOWER
I20260925 02:44:32.070161 12860 consensus_queue.cc:260] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:32.070694 12859 raft_consensus.cc:348] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:32.071164 12859 raft_consensus.cc:374] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:32.071282 12859 raft_consensus.cc:729] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb3f982ef3a4a64a3ecf29fdea39345, State: Initialized, Role: FOLLOWER
I20260925 02:44:32.072065 12860 ts_tablet_manager.cc:1428] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 02:44:32.072011 12859 consensus_queue.cc:260] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
W20260925 02:44:32.075383 12537 tablet.cc:2369] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:44:32.078413 12859 ts_tablet_manager.cc:1428] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: Time spent starting tablet: real 0.020s user 0.011s sys 0.004s
I20260925 02:44:32.078830 12682 heartbeater.cc:503] Master 127.5.56.190:42051 was elected leader, sending a full tablet report...
I20260925 02:44:32.082262 12863 raft_consensus.cc:348] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:32.082834 12863 raft_consensus.cc:374] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:32.082932 12863 raft_consensus.cc:729] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19a41448a394698b14bfc67ffd20931, State: Initialized, Role: FOLLOWER
I20260925 02:44:32.083462 12863 consensus_queue.cc:260] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:32.084538 12823 heartbeater.cc:503] Master 127.5.56.190:42051 was elected leader, sending a full tablet report...
I20260925 02:44:32.085271 12863 ts_tablet_manager.cc:1428] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260925 02:44:32.150439 12828 tablet.cc:2369] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:44:32.276494 12690 tablet.cc:2369] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:44:32.332211 12866 raft_consensus.cc:482] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:44:32.332502 12866 raft_consensus.cc:504] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:32.334281 12866 leader_election.cc:288] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d19a41448a394698b14bfc67ffd20931 (127.5.56.131:34367), feb3f982ef3a4a64a3ecf29fdea39345 (127.5.56.130:35741)
I20260925 02:44:32.344079 12855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:32.334862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:38663 (local address 127.5.56.131:34367)
0925 02:44:32.335080 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:32.335086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:32.335826 (+ 740us) server_negotiation.cc:407] Connection header received
0925 02:44:32.336031 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:32.336039 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:32.336127 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:32.336259 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:32.337148 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.338255 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.341506 (+ 3251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.342805 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.342998 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.343082 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:32.343645 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 02:44:32.343792 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 02:44:32.344079 12856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:32.334815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:43027 (local address 127.5.56.130:35741)
0925 02:44:32.335028 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:32.335034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:32.335826 (+ 792us) server_negotiation.cc:407] Connection header received
0925 02:44:32.336112 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:32.336117 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:32.336200 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:32.336291 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:32.337158 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.338235 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.341506 (+ 3271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.342820 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.342998 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.343082 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:32.343645 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 02:44:32.343790 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:44:32.344336 12870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:32.334768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:34367 (local address 127.5.56.129:38663)
0925 02:44:32.335682 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:32.335704 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:32.335889 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:32.336295 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:32.336301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:32.336341 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:32.336909 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:32.336922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.338431 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.338437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:32.341276 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:32.341301 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.343091 (+ 1790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.343099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:32.343382 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.343482 (+ 100us) client_negotiation.cc:769] Sending connection context
0925 02:44:32.343594 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:44:32.343790 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"mutex_wait_us":314,"spinlock_wait_cycles":18048,"thread_start_us":423,"threads_started":1}
I20260925 02:44:32.344769 12869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:32.334768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:35741 (local address 127.5.56.129:43027)
0925 02:44:32.335683 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:32.335704 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:32.335889 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:32.336295 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:32.336301 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:32.336337 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:32.336916 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:32.336924 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.338431 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.338437 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:32.341276 (+ 2839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:32.341301 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.343106 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.343111 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:32.343383 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.343487 (+ 104us) client_negotiation.cc:769] Sending connection context
0925 02:44:32.343595 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:44:32.343790 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"mutex_wait_us":411,"spinlock_wait_cycles":9728,"thread_start_us":546,"threads_started":1}
I20260925 02:44:32.345641 12777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ee5c20c3c4625ab62d53fe5d82740" candidate_uuid: "c088d4b02c704a7ba40c01da541a3fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19a41448a394698b14bfc67ffd20931" is_pre_election: true
I20260925 02:44:32.345755 12635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ee5c20c3c4625ab62d53fe5d82740" candidate_uuid: "c088d4b02c704a7ba40c01da541a3fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" is_pre_election: true
I20260925 02:44:32.346264 12777 raft_consensus.cc:2454] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c088d4b02c704a7ba40c01da541a3fa8 in term 0.
I20260925 02:44:32.346264 12635 raft_consensus.cc:2454] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c088d4b02c704a7ba40c01da541a3fa8 in term 0.
I20260925 02:44:32.347013 12421 leader_election.cc:302] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c088d4b02c704a7ba40c01da541a3fa8, d19a41448a394698b14bfc67ffd20931; no voters:
I20260925 02:44:32.347395 12866 raft_consensus.cc:2781] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:44:32.347581 12866 raft_consensus.cc:482] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:44:32.347699 12866 raft_consensus.cc:3037] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:32.350162 12866 raft_consensus.cc:504] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:32.350914 12866 leader_election.cc:288] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [CANDIDATE]: Term 1 election: Requested vote from peers d19a41448a394698b14bfc67ffd20931 (127.5.56.131:34367), feb3f982ef3a4a64a3ecf29fdea39345 (127.5.56.130:35741)
I20260925 02:44:32.351464 12635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ee5c20c3c4625ab62d53fe5d82740" candidate_uuid: "c088d4b02c704a7ba40c01da541a3fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb3f982ef3a4a64a3ecf29fdea39345"
I20260925 02:44:32.351464 12777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235ee5c20c3c4625ab62d53fe5d82740" candidate_uuid: "c088d4b02c704a7ba40c01da541a3fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19a41448a394698b14bfc67ffd20931"
I20260925 02:44:32.351698 12777 raft_consensus.cc:3037] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:32.351698 12635 raft_consensus.cc:3037] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:32.355147 12777 raft_consensus.cc:2454] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c088d4b02c704a7ba40c01da541a3fa8 in term 1.
I20260925 02:44:32.355162 12635 raft_consensus.cc:2454] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c088d4b02c704a7ba40c01da541a3fa8 in term 1.
I20260925 02:44:32.355772 12422 leader_election.cc:302] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: c088d4b02c704a7ba40c01da541a3fa8, d19a41448a394698b14bfc67ffd20931, feb3f982ef3a4a64a3ecf29fdea39345; no voters:
I20260925 02:44:32.356155 12866 raft_consensus.cc:2781] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:32.356594 12866 raft_consensus.cc:686] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 1 LEADER]: Becoming Leader. State: Replica: c088d4b02c704a7ba40c01da541a3fa8, State: Running, Role: LEADER
I20260925 02:44:32.357195 12866 consensus_queue.cc:237] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:32.362790 11556 catalog_manager.cc:5495] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c088d4b02c704a7ba40c01da541a3fa8 (127.5.56.129). New cstate: current_term: 1 leader_uuid: "c088d4b02c704a7ba40c01da541a3fa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } health_report { overall_health: HEALTHY } } }
I20260925 02:44:32.399910 12858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:32.393294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36200 (local address 127.5.56.129:46809)
0925 02:44:32.393530 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:32.393536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:32.393716 (+ 180us) server_negotiation.cc:407] Connection header received
0925 02:44:32.394044 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:32.394050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:32.394225 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:32.394421 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:44:32.394986 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.395927 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.397679 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.398113 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.398224 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.398776 (+ 552us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:44:32.399319 (+ 543us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:44:32.399437 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:32.399522 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 02:44:32.399600 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 02:44:32.410321 12635 raft_consensus.cc:1261] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345 [term 1 FOLLOWER]: Refusing update from remote peer c088d4b02c704a7ba40c01da541a3fa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:32.410321 12777 raft_consensus.cc:1261] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931 [term 1 FOLLOWER]: Refusing update from remote peer c088d4b02c704a7ba40c01da541a3fa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:32.411779 12871 consensus_queue.cc:1035] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:32.412243 12866 consensus_queue.cc:1035] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:32.427601 12875 mvcc.cc:205] Tried to move back new op lower bound from 7333086299779223552 to 7333086299579383808. Current Snapshot: MvccSnapshot[applied={T|T < 7333086299779223552}]
I20260925 02:44:32.466424 11555 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:32.466782 11555 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35368:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:32.474534 11555 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d65122eeada4f58aa3c9176c0df7ba1: Sending DeleteTablet for 3 replicas of tablet 235ee5c20c3c4625ab62d53fe5d82740
I20260925 02:44:32.475741 12757 tablet_service.cc:1514] Processing DeleteTablet for tablet 235ee5c20c3c4625ab62d53fe5d82740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:32 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:41718
I20260925 02:44:32.476111 12468 tablet_service.cc:1514] Processing DeleteTablet for tablet 235ee5c20c3c4625ab62d53fe5d82740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:32 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:36190
I20260925 02:44:32.476102 12615 tablet_service.cc:1514] Processing DeleteTablet for tablet 235ee5c20c3c4625ab62d53fe5d82740 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:32 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:56440
I20260925 02:44:32.488662 12887 tablet_replica.cc:330] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: stopping tablet replica
I20260925 02:44:32.489531 12887 raft_consensus.cc:2229] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:44:32.490002 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 11524
I20260925 02:44:32.490660 12887 raft_consensus.cc:2258] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:32.491840 11586 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:32.493803 12887 ts_tablet_manager.cc:1905] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:32.494591 12886 tablet_replica.cc:330] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: stopping tablet replica
I20260925 02:44:32.495954 12885 tablet_replica.cc:330] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: stopping tablet replica
I20260925 02:44:32.496455 12885 raft_consensus.cc:2229] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:32.496464 12886 raft_consensus.cc:2229] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:32.496982 12885 raft_consensus.cc:2258] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:32.496977 12886 raft_consensus.cc:2258] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:32.506186 12885 ts_tablet_manager.cc:1905] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:32.506255 12886 ts_tablet_manager.cc:1905] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:32.512305 12885 ts_tablet_manager.cc:1918] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:32.512547 12885 log.cc:1196] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/235ee5c20c3c4625ab62d53fe5d82740
I20260925 02:44:32.513175 12885 ts_tablet_manager.cc:1939] T 235ee5c20c3c4625ab62d53fe5d82740 P d19a41448a394698b14bfc67ffd20931: Deleting consensus metadata
I20260925 02:44:32.524349 12887 ts_tablet_manager.cc:1918] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:32.524626 12887 log.cc:1196] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/235ee5c20c3c4625ab62d53fe5d82740
I20260925 02:44:32.525264 12887 ts_tablet_manager.cc:1939] T 235ee5c20c3c4625ab62d53fe5d82740 P c088d4b02c704a7ba40c01da541a3fa8: Deleting consensus metadata
I20260925 02:44:32.528575 12886 ts_tablet_manager.cc:1918] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:32.528787 12886 log.cc:1196] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/235ee5c20c3c4625ab62d53fe5d82740
I20260925 02:44:32.529383 12886 ts_tablet_manager.cc:1939] T 235ee5c20c3c4625ab62d53fe5d82740 P feb3f982ef3a4a64a3ecf29fdea39345: Deleting consensus metadata
I20260925 02:44:32.530920 12889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:32.478793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38998 (local address 127.5.56.189:43763)
0925 02:44:32.497900 (+ 19107us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:32.497911 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:32.497947 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:44:32.498038 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:32.498046 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:32.498172 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:32.498359 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:32.500621 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.502206 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.504690 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.511547 (+ 6857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.511787 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.516051 (+ 4264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:32.516059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:32.516065 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:32.516115 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:32.519975 (+ 3860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:32.521060 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:32.521066 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:32.521070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:32.521231 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:32.521728 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:32.521733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:32.521736 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:32.522150 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:32.529700 (+ 7550us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:32.529879 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 02:44:32.530534 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18916,"thread_start_us":19725,"threads_started":1}
I20260925 02:44:32.544744 12890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:32.486300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.5.56.188:41937)
0925 02:44:32.489803 (+ 3503us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:32.489809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:32.524334 (+ 34525us) server_negotiation.cc:407] Connection header received
0925 02:44:32.524900 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:32.524908 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:32.525016 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:32.525120 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:32.527261 (+ 2141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.528411 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.532312 (+ 3901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.532988 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.533111 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.536893 (+ 3782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:32.536901 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:32.536906 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:32.536955 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:32.540861 (+ 3906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:32.541961 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:32.541968 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:32.541971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:32.542154 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:32.542750 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:32.542758 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:32.542763 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:32.543240 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:32.543682 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:32.544067 (+ 385us) server_negotiation.cc:299] Negotiation successful
0925 02:44:32.544331 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3360,"thread_start_us":4320,"threads_started":1}
I20260925 02:44:32.668509 11541 catalog_manager.cc:4841] TS feb3f982ef3a4a64a3ecf29fdea39345 (127.5.56.130:35741): tablet 235ee5c20c3c4625ab62d53fe5d82740 (table test-table [id=fbfe091b7733437d94c3163e52368f55]) successfully deleted
I20260925 02:44:32.670964 11540 catalog_manager.cc:4841] TS d19a41448a394698b14bfc67ffd20931 (127.5.56.131:34367): tablet 235ee5c20c3c4625ab62d53fe5d82740 (table test-table [id=fbfe091b7733437d94c3163e52368f55]) successfully deleted
W20260925 02:44:32.671255 11540 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.5.56.131:34367, user_credentials={real_user=slave}} blocked reactor thread for 155900us
I20260925 02:44:32.672308 11542 catalog_manager.cc:4841] TS c088d4b02c704a7ba40c01da541a3fa8 (127.5.56.129:46809): tablet 235ee5c20c3c4625ab62d53fe5d82740 (table test-table [id=fbfe091b7733437d94c3163e52368f55]) successfully deleted
I20260925 02:44:32.678838 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 11524
I20260925 02:44:32.724321 12889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:32.706779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39002 (local address 127.5.56.189:43763)
0925 02:44:32.707035 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:32.707045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:32.708113 (+ 1068us) server_negotiation.cc:407] Connection header received
0925 02:44:32.708405 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:32.708413 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:32.708528 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:32.708616 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:32.711011 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.712132 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.714206 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.714900 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.715041 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.718317 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:32.718322 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:32.718325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:32.718360 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:32.721291 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:32.722036 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:32.722042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:32.722045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:32.722198 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:32.722661 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:32.722667 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:32.722671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:32.723111 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:32.723374 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:32.723744 (+ 370us) server_negotiation.cc:299] Negotiation successful
0925 02:44:32.723998 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:44:32.725455 12890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:32.707829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.5.56.188:41937)
0925 02:44:32.708021 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:32.708028 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:32.708933 (+ 905us) server_negotiation.cc:407] Connection header received
0925 02:44:32.709089 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:32.709096 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:32.709181 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:32.709292 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:32.711350 (+ 2058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.712056 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.713476 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:32.713924 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:32.714070 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:32.717934 (+ 3864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:32.717941 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:32.717946 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:32.717991 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:32.721790 (+ 3799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:32.722511 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:32.722518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:32.722529 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:32.722676 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:32.723127 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:32.723132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:32.723136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:32.723510 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:32.723764 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:32.724871 (+ 1107us) server_negotiation.cc:299] Negotiation successful
0925 02:44:32.725112 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:44:33.440728 12898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:33.439842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.131:56943)
0925 02:44:33.440332 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:33.440493 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":245,"threads_started":1}
W20260925 02:44:33.441286 12823 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:42051 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
I20260925 02:44:33.461218 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:33.460283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.130:41909)
0925 02:44:33.460772 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:33.460974 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":554,"threads_started":1}
I20260925 02:44:33.461370 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:33.460417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.129:33919)
0925 02:44:33.460922 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:33.461114 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":282,"threads_started":1}
W20260925 02:44:33.461808 12682 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:42051 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
W20260925 02:44:33.461808 12535 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:42051 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
I20260925 02:44:33.975015 12901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d65122eeada4f58aa3c9176c0df7ba1)
I20260925 02:44:33.975307 12902 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3d65122eeada4f58aa3c9176c0df7ba1)
I20260925 02:44:33.975435 12901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } }
I20260925 02:44:33.975638 12902 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } }
I20260925 02:44:33.983388 12902 leader_election.cc:288] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051), 5701926ddb054d9cbc2012bf74a7c157 (127.5.56.189:43763)
I20260925 02:44:33.983675 12901 leader_election.cc:288] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051), fbfd3860e2ff46929abee5d1d5212edf (127.5.56.188:41937)
I20260925 02:44:33.988142 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:33.977317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35878)
0925 02:44:33.987401 (+ 10084us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:33.987661 (+ 260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":9906,"thread_start_us":9676,"threads_started":1}
I20260925 02:44:33.988798 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:33.977317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35876)
0925 02:44:33.987524 (+ 10207us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:33.987731 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10059,"threads_started":1}
W20260925 02:44:33.989169 12155 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
W20260925 02:44:33.989225 12341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
I20260925 02:44:33.997690 12904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:33.997200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35906)
0925 02:44:33.997371 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:33.997490 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260925 02:44:33.997960 12341 leader_election.cc:334] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051): Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
I20260925 02:44:33.998312 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:33.988791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:41937 (local address 127.0.0.1:52702)
0925 02:44:33.989108 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:33.989126 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:33.989276 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:33.989822 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:33.989829 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:33.989879 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:33.990177 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:33.990195 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:33.992340 (+ 2145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:33.992349 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:33.995259 (+ 2910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:33.995283 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:33.997494 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:33.997500 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:33.997699 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:33.997744 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:44:33.997844 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:44:33.998003 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260925 02:44:33.998333 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:33.977499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:43763 (local address 127.0.0.1:39010)
0925 02:44:33.988532 (+ 11033us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:33.988561 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:33.988747 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:33.989463 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:33.989467 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:33.989499 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:33.990095 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:33.990114 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:33.992335 (+ 2221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:33.992343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:33.995362 (+ 3019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:33.995385 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:33.997502 (+ 2117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:33.997510 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:33.997691 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:33.997717 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:44:33.997838 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 02:44:33.997991 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10852,"thread_start_us":10687,"threads_started":1}
I20260925 02:44:33.998939 12906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:33.977524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.5.56.189:43763)
0925 02:44:33.987061 (+ 9537us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:33.987068 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:33.988718 (+ 1650us) server_negotiation.cc:407] Connection header received
0925 02:44:33.988940 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:33.988945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:33.989020 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:33.989181 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:33.990424 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:33.992038 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:33.995641 (+ 3603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:33.997135 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:33.997265 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:33.997365 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:33.997992 (+ 627us) server_negotiation.cc:299] Negotiation successful
0925 02:44:33.998149 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9360,"thread_start_us":9113,"threads_started":1}
I20260925 02:44:33.999053 12907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:33.987749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52702 (local address 127.5.56.188:41937)
0925 02:44:33.988863 (+ 1114us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:33.988870 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:33.989416 (+ 546us) server_negotiation.cc:407] Connection header received
0925 02:44:33.989481 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:33.989487 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:33.989598 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:33.989705 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:33.990436 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:33.992135 (+ 1699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:33.995546 (+ 3411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:33.997137 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:33.997268 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:33.997369 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:33.998541 (+ 1172us) server_negotiation.cc:299] Negotiation successful
0925 02:44:33.998690 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":978,"thread_start_us":819,"threads_started":1}
I20260925 02:44:33.999571 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:33.998541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35922)
0925 02:44:33.999296 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:33.999417 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":656}
I20260925 02:44:33.999598 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbfd3860e2ff46929abee5d1d5212edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5701926ddb054d9cbc2012bf74a7c157" is_pre_election: true
I20260925 02:44:33.999693 12365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5701926ddb054d9cbc2012bf74a7c157" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbfd3860e2ff46929abee5d1d5212edf" is_pre_election: true
W20260925 02:44:33.999781 12155 leader_election.cc:334] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051): Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
I20260925 02:44:34.000001 12178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbfd3860e2ff46929abee5d1d5212edf in term 1.
I20260925 02:44:34.000027 12365 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5701926ddb054d9cbc2012bf74a7c157 in term 1.
I20260925 02:44:34.000511 12155 leader_election.cc:302] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5701926ddb054d9cbc2012bf74a7c157, fbfd3860e2ff46929abee5d1d5212edf; no voters: 3d65122eeada4f58aa3c9176c0df7ba1
I20260925 02:44:34.000527 12338 leader_election.cc:302] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5701926ddb054d9cbc2012bf74a7c157, fbfd3860e2ff46929abee5d1d5212edf; no voters: 3d65122eeada4f58aa3c9176c0df7ba1
I20260925 02:44:34.000867 12901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:44:34.000900 12902 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 02:44:34.000991 12901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d65122eeada4f58aa3c9176c0df7ba1)
I20260925 02:44:34.001017 12902 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3d65122eeada4f58aa3c9176c0df7ba1)
I20260925 02:44:34.001087 12901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:44:34.001106 12902 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:44:34.003360 12902 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } }
I20260925 02:44:34.003461 12901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } }
I20260925 02:44:34.004179 12902 leader_election.cc:288] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 2 election: Requested vote from peers 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051), 5701926ddb054d9cbc2012bf74a7c157 (127.5.56.189:43763)
I20260925 02:44:34.004298 12901 leader_election.cc:288] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [CANDIDATE]: Term 2 election: Requested vote from peers 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051), fbfd3860e2ff46929abee5d1d5212edf (127.5.56.188:41937)
I20260925 02:44:34.004739 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbfd3860e2ff46929abee5d1d5212edf" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5701926ddb054d9cbc2012bf74a7c157"
I20260925 02:44:34.004897 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:34.004400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35948)
0925 02:44:34.004603 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:34.004699 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 02:44:34.004961 12365 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5701926ddb054d9cbc2012bf74a7c157" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fbfd3860e2ff46929abee5d1d5212edf"
I20260925 02:44:34.005141 12178 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate fbfd3860e2ff46929abee5d1d5212edf in current term 2: Already voted for candidate 5701926ddb054d9cbc2012bf74a7c157 in this term.
I20260925 02:44:34.005223 12365 raft_consensus.cc:2379] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5701926ddb054d9cbc2012bf74a7c157 in current term 2: Already voted for candidate fbfd3860e2ff46929abee5d1d5212edf in this term.
I20260925 02:44:34.005259 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:34.004698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35938)
0925 02:44:34.004950 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:34.005099 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 02:44:34.006445 12905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:34.006110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35958)
0925 02:44:34.006237 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:34.006308 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:44:34.006623 12341 leader_election.cc:334] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051): Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
I20260925 02:44:34.006764 12341 leader_election.cc:302] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fbfd3860e2ff46929abee5d1d5212edf; no voters: 3d65122eeada4f58aa3c9176c0df7ba1, 5701926ddb054d9cbc2012bf74a7c157
I20260925 02:44:34.007062 12902 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 02:44:34.008046 12903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:34.007695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:35966)
0925 02:44:34.007846 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:34.007917 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:44:34.008232 12155 leader_election.cc:334] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051): Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
I20260925 02:44:34.008379 12155 leader_election.cc:302] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5701926ddb054d9cbc2012bf74a7c157; no voters: 3d65122eeada4f58aa3c9176c0df7ba1, fbfd3860e2ff46929abee5d1d5212edf
I20260925 02:44:34.008649 12901 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 02:44:34.442962 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:34.442073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.131:35313)
0925 02:44:34.442592 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:34.442727 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":214,"threads_started":1}
I20260925 02:44:34.463198 12911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:34.462500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.130:55155)
0925 02:44:34.462933 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:34.463019 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20260925 02:44:34.463565 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:34.462499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.129:43397)
0925 02:44:34.463251 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:34.463356 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":541,"threads_started":1}
I20260925 02:44:35.444670 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:35.443832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.131:57201)
0925 02:44:35.444349 (+ 517us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:35.444449 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":290,"threads_started":1}
I20260925 02:44:35.464939 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:35.464099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.130:51659)
0925 02:44:35.464628 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:35.464710 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":327,"threads_started":1}
I20260925 02:44:35.465255 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:35.464393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.129:45729)
0925 02:44:35.464944 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:35.465021 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":307,"threads_started":1}
I20260925 02:44:36.020718 12916 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:44:36.021003 12916 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } }
I20260925 02:44:36.021720 12916 leader_election.cc:288] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051), 5701926ddb054d9cbc2012bf74a7c157 (127.5.56.189:43763)
I20260925 02:44:36.022264 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbfd3860e2ff46929abee5d1d5212edf" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5701926ddb054d9cbc2012bf74a7c157" is_pre_election: true
I20260925 02:44:36.022560 12178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbfd3860e2ff46929abee5d1d5212edf in term 2.
I20260925 02:44:36.022720 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.021785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:36266)
0925 02:44:36.022320 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.022473 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":263,"threads_started":1}
I20260925 02:44:36.023013 12338 leader_election.cc:302] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5701926ddb054d9cbc2012bf74a7c157, fbfd3860e2ff46929abee5d1d5212edf; no voters:
I20260925 02:44:36.023343 12916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 02:44:36.023481 12916 raft_consensus.cc:482] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:44:36.023602 12916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:44:36.024210 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.023817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:36282)
0925 02:44:36.023978 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.024069 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 02:44:36.024400 12341 leader_election.cc:334] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051): Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
I20260925 02:44:36.025938 12916 raft_consensus.cc:504] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } }
I20260925 02:44:36.026599 12916 leader_election.cc:288] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 3 election: Requested vote from peers 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051), 5701926ddb054d9cbc2012bf74a7c157 (127.5.56.189:43763)
I20260925 02:44:36.027064 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.026645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:36284)
0925 02:44:36.026855 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.026930 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260925 02:44:36.027048 12178 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fbfd3860e2ff46929abee5d1d5212edf" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "5701926ddb054d9cbc2012bf74a7c157"
I20260925 02:44:36.027237 12178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:44:36.027948 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.027643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:36288)
0925 02:44:36.027778 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.027849 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 02:44:36.028090 12341 leader_election.cc:334] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051): Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
I20260925 02:44:36.029467 12178 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbfd3860e2ff46929abee5d1d5212edf in term 3.
I20260925 02:44:36.029886 12338 leader_election.cc:302] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5701926ddb054d9cbc2012bf74a7c157, fbfd3860e2ff46929abee5d1d5212edf; no voters: 3d65122eeada4f58aa3c9176c0df7ba1
I20260925 02:44:36.030166 12916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:44:36.030654 12916 raft_consensus.cc:686] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [term 3 LEADER]: Becoming Leader. State: Replica: fbfd3860e2ff46929abee5d1d5212edf, State: Running, Role: LEADER
I20260925 02:44:36.031039 12916 consensus_queue.cc:237] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } }
I20260925 02:44:36.033066 12919 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbfd3860e2ff46929abee5d1d5212edf. Latest consensus state: current_term: 3 leader_uuid: "fbfd3860e2ff46929abee5d1d5212edf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:36.033511 12919 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:36.034237 12921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:36.038331 12921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:36.039323 12921 catalog_manager.cc:1259] Loaded cluster ID: 714faf126fea4489854ae85ccb24415d
I20260925 02:44:36.039404 12921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:36.040588 12921 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:36.041445 12921 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: Loaded TSK: 0
I20260925 02:44:36.042449 12921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:44:36.108276 12354 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52692:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:44:36.115597 12354 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:44:36.119890 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.119434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:36318)
0925 02:44:36.119637 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.119741 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260925 02:44:36.120056 12178 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [term 3 FOLLOWER]: Refusing update from remote peer fbfd3860e2ff46929abee5d1d5212edf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 02:44:36.120837 12919 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [LEADER]: Connected to new peer: Peer: permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260925 02:44:36.121078 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.120598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:36320)
0925 02:44:36.120797 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.120895 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 02:44:36.121253 12341 consensus_peers.cc:487] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf -> Peer 3d65122eeada4f58aa3c9176c0df7ba1 (127.5.56.190:42051): Couldn't send request to peer 3d65122eeada4f58aa3c9176c0df7ba1. Status: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 02:44:36.123641 12923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fbfd3860e2ff46929abee5d1d5212edf. Latest consensus state: current_term: 3 leader_uuid: "fbfd3860e2ff46929abee5d1d5212edf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:36.124019 12923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:44:36.125468 12919 sys_catalog.cc:455] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "fbfd3860e2ff46929abee5d1d5212edf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:36.125780 12919 sys_catalog.cc:458] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:36.126340 12924 mvcc.cc:205] Tried to move back new op lower bound from 7333086314980184064 to 7333086314626813952. Current Snapshot: MvccSnapshot[applied={T|T < 7333086314980184064}]
I20260925 02:44:36.127111 12923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "fbfd3860e2ff46929abee5d1d5212edf" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "3d65122eeada4f58aa3c9176c0df7ba1" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42051 } } peers { permanent_uuid: "5701926ddb054d9cbc2012bf74a7c157" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 43763 } attrs { promote: false } } peers { permanent_uuid: "fbfd3860e2ff46929abee5d1d5212edf" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 41937 } attrs { promote: false } } }
I20260925 02:44:36.127398 12923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5701926ddb054d9cbc2012bf74a7c157 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:44:36.152307 12930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.135011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41730 (local address 127.5.56.131:34367)
0925 02:44:36.144978 (+ 9967us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:36.144983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:36.145001 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:44:36.145064 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:36.145067 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:36.145129 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:36.145262 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:36.146141 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.147304 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.150441 (+ 3137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.151315 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.151462 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:36.151572 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:36.151859 (+ 287us) server_negotiation.cc:299] Negotiation successful
0925 02:44:36.151999 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9836,"thread_start_us":9732,"threads_started":1}
I20260925 02:44:36.152458 12929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.134644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:34367 (local address 127.0.0.1:41730)
0925 02:44:36.135372 (+ 728us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.135395 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:36.135526 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:36.145283 (+ 9757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:36.145288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:36.145317 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:36.145853 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:36.145867 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.147476 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.147485 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:36.149468 (+ 1983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:36.149483 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.151487 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.151493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:36.151676 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:36.151700 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:44:36.151830 (+ 130us) client_negotiation.cc:240] Negotiation successful
0925 02:44:36.151980 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"mutex_wait_us":86,"thread_start_us":500,"threads_started":1}
I20260925 02:44:36.152829 12917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.134136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:35741 (local address 127.0.0.1:56444)
0925 02:44:36.134295 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.134319 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:36.134478 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:36.145326 (+ 10848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:36.145331 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:36.145357 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:36.145879 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:36.145931 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.147587 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.147591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:36.150124 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:36.150141 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.152118 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.152127 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:36.152292 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:36.152320 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:44:36.152374 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 02:44:36.152510 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":85}
I20260925 02:44:36.153396 12757 tablet_service.cc:1467] Processing CreateTablet for tablet ff381d4b14014cf5984c4d3866d65404 (DEFAULT_TABLE table=test-table [id=26033ac981ce4d9a8aadc62ac768dbe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:36.153800 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.134265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56444 (local address 127.5.56.130:35741)
0925 02:44:36.144943 (+ 10678us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:36.144950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:36.144972 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:44:36.145065 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:36.145070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:36.145153 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:36.146102 (+ 949us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:36.146316 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.147353 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.150515 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.151870 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.153212 (+ 1342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:36.153317 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:36.153366 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 02:44:36.153511 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10530,"thread_start_us":10879,"threads_started":1}
I20260925 02:44:36.153872 12757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff381d4b14014cf5984c4d3866d65404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:36.158836 12931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.135704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46809 (local address 127.0.0.1:36216)
0925 02:44:36.136350 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.136365 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:36.136477 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:36.145758 (+ 9281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:36.145765 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:36.145804 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:36.146359 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:36.146369 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.147508 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.147514 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:36.150165 (+ 2651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:36.150176 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.151865 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.151871 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:36.152112 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:36.152149 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:44:36.152215 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 02:44:36.158546 (+ 6331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"mutex_wait_us":181,"thread_start_us":736,"threads_started":1}
I20260925 02:44:36.159513 12615 tablet_service.cc:1467] Processing CreateTablet for tablet ff381d4b14014cf5984c4d3866d65404 (DEFAULT_TABLE table=test-table [id=26033ac981ce4d9a8aadc62ac768dbe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:36.159634 12933 tablet_bootstrap.cc:492] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931: Bootstrap starting.
I20260925 02:44:36.160017 12615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff381d4b14014cf5984c4d3866d65404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:36.161432 12933 tablet_bootstrap.cc:654] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:36.167228 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.136232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.5.56.129:46809)
0925 02:44:36.145399 (+ 9167us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:36.145408 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:36.145428 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:44:36.145499 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:36.145507 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:36.145601 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:36.145712 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:36.146573 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.147338 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.150316 (+ 2978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.151629 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.166256 (+ 14627us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:36.166426 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:36.166505 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 02:44:36.166685 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9039,"thread_start_us":8232,"threads_started":1}
I20260925 02:44:36.168249 12468 tablet_service.cc:1467] Processing CreateTablet for tablet ff381d4b14014cf5984c4d3866d65404 (DEFAULT_TABLE table=test-table [id=26033ac981ce4d9a8aadc62ac768dbe5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:36.168881 12468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff381d4b14014cf5984c4d3866d65404. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:36.183224 12935 tablet_bootstrap.cc:492] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345: Bootstrap starting.
I20260925 02:44:36.184506 12936 tablet_bootstrap.cc:492] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8: Bootstrap starting.
I20260925 02:44:36.185009 12933 tablet_bootstrap.cc:492] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931: No bootstrap required, opened a new log
I20260925 02:44:36.185189 12933 ts_tablet_manager.cc:1397] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931: Time spent bootstrapping tablet: real 0.026s user 0.015s sys 0.008s
I20260925 02:44:36.185395 12935 tablet_bootstrap.cc:654] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:36.186151 12933 raft_consensus.cc:348] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:36.186501 12933 raft_consensus.cc:374] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:36.186580 12936 tablet_bootstrap.cc:654] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:36.186640 12933 raft_consensus.cc:729] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d19a41448a394698b14bfc67ffd20931, State: Initialized, Role: FOLLOWER
I20260925 02:44:36.186911 12933 consensus_queue.cc:260] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:36.188824 12933 ts_tablet_manager.cc:1428] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260925 02:44:36.189381 12936 tablet_bootstrap.cc:492] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8: No bootstrap required, opened a new log
I20260925 02:44:36.189605 12936 ts_tablet_manager.cc:1397] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260925 02:44:36.190555 12935 tablet_bootstrap.cc:492] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345: No bootstrap required, opened a new log
I20260925 02:44:36.190759 12935 ts_tablet_manager.cc:1397] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20260925 02:44:36.190598 12936 raft_consensus.cc:348] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:36.190997 12936 raft_consensus.cc:374] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:36.191196 12936 raft_consensus.cc:729] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c088d4b02c704a7ba40c01da541a3fa8, State: Initialized, Role: FOLLOWER
I20260925 02:44:36.191519 12936 consensus_queue.cc:260] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:36.191769 12935 raft_consensus.cc:348] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:36.192024 12935 raft_consensus.cc:374] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:36.192113 12822 heartbeater.cc:503] Master 127.5.56.188:41937 was elected leader, sending a full tablet report...
I20260925 02:44:36.192131 12935 raft_consensus.cc:729] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feb3f982ef3a4a64a3ecf29fdea39345, State: Initialized, Role: FOLLOWER
I20260925 02:44:36.192375 12935 consensus_queue.cc:260] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:36.193251 12941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.188115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.131:40285)
0925 02:44:36.192706 (+ 4591us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.192857 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4414,"thread_start_us":320,"threads_started":1}
I20260925 02:44:36.195382 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.194116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.131:52657)
0925 02:44:36.194802 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.194982 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":348,"threads_started":1}
I20260925 02:44:36.195801 12935 ts_tablet_manager.cc:1428] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260925 02:44:36.197510 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.196289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.130:39331)
0925 02:44:36.197088 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.197230 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":661,"threads_started":1}
I20260925 02:44:36.199522 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.198574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.130:51205)
0925 02:44:36.199284 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.199372 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":599}
I20260925 02:44:36.199738 12681 heartbeater.cc:503] Master 127.5.56.188:41937 was elected leader, sending a full tablet report...
I20260925 02:44:36.202358 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.200746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.129:37115)
0925 02:44:36.201333 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.201479 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":333,"threads_started":1}
I20260925 02:44:36.202330 12936 ts_tablet_manager.cc:1428] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8: Time spent starting tablet: real 0.011s user 0.006s sys 0.003s
I20260925 02:44:36.203155 12534 heartbeater.cc:503] Master 127.5.56.188:41937 was elected leader, sending a full tablet report...
I20260925 02:44:36.204994 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.203471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.129:41485)
0925 02:44:36.203834 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.204822 (+ 988us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257}
I20260925 02:44:36.307783 12942 raft_consensus.cc:482] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:44:36.308058 12942 raft_consensus.cc:504] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:36.309008 12942 leader_election.cc:288] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feb3f982ef3a4a64a3ecf29fdea39345 (127.5.56.130:35741), d19a41448a394698b14bfc67ffd20931 (127.5.56.131:34367)
I20260925 02:44:36.309691 12635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff381d4b14014cf5984c4d3866d65404" candidate_uuid: "c088d4b02c704a7ba40c01da541a3fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" is_pre_election: true
I20260925 02:44:36.309948 12635 raft_consensus.cc:2454] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c088d4b02c704a7ba40c01da541a3fa8 in term 0.
I20260925 02:44:36.310513 12422 leader_election.cc:302] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c088d4b02c704a7ba40c01da541a3fa8, feb3f982ef3a4a64a3ecf29fdea39345; no voters:
I20260925 02:44:36.310902 12942 raft_consensus.cc:2781] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:44:36.311048 12942 raft_consensus.cc:482] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:44:36.311003 12777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff381d4b14014cf5984c4d3866d65404" candidate_uuid: "c088d4b02c704a7ba40c01da541a3fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19a41448a394698b14bfc67ffd20931" is_pre_election: true
I20260925 02:44:36.311163 12942 raft_consensus.cc:3037] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:36.311282 12777 raft_consensus.cc:2454] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c088d4b02c704a7ba40c01da541a3fa8 in term 0.
I20260925 02:44:36.314646 12942 raft_consensus.cc:504] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:36.315372 12942 leader_election.cc:288] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [CANDIDATE]: Term 1 election: Requested vote from peers feb3f982ef3a4a64a3ecf29fdea39345 (127.5.56.130:35741), d19a41448a394698b14bfc67ffd20931 (127.5.56.131:34367)
I20260925 02:44:36.316073 12777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff381d4b14014cf5984c4d3866d65404" candidate_uuid: "c088d4b02c704a7ba40c01da541a3fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d19a41448a394698b14bfc67ffd20931"
I20260925 02:44:36.316130 12635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff381d4b14014cf5984c4d3866d65404" candidate_uuid: "c088d4b02c704a7ba40c01da541a3fa8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feb3f982ef3a4a64a3ecf29fdea39345"
I20260925 02:44:36.316298 12777 raft_consensus.cc:3037] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:36.316331 12635 raft_consensus.cc:3037] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:36.319409 12777 raft_consensus.cc:2454] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c088d4b02c704a7ba40c01da541a3fa8 in term 1.
I20260925 02:44:36.319409 12635 raft_consensus.cc:2454] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c088d4b02c704a7ba40c01da541a3fa8 in term 1.
I20260925 02:44:36.319846 12422 leader_election.cc:302] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c088d4b02c704a7ba40c01da541a3fa8, feb3f982ef3a4a64a3ecf29fdea39345; no voters:
I20260925 02:44:36.320134 12942 raft_consensus.cc:2781] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:36.320573 12942 raft_consensus.cc:686] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 1 LEADER]: Becoming Leader. State: Replica: c088d4b02c704a7ba40c01da541a3fa8, State: Running, Role: LEADER
I20260925 02:44:36.320879 12942 consensus_queue.cc:237] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } } peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } }
I20260925 02:44:36.321839 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.321383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.129:46119)
0925 02:44:36.321552 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.321655 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 02:44:36.324010 12354 catalog_manager.cc:5495] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c088d4b02c704a7ba40c01da541a3fa8 (127.5.56.129). New cstate: current_term: 1 leader_uuid: "c088d4b02c704a7ba40c01da541a3fa8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c088d4b02c704a7ba40c01da541a3fa8" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46809 } health_report { overall_health: HEALTHY } } }
I20260925 02:44:36.382645 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.377831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.5.56.129:46809)
0925 02:44:36.378075 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:36.378081 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:36.378101 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:44:36.378281 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:36.378285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:36.378349 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:36.378497 (+ 148us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:44:36.378897 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.379606 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.380929 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.381350 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.381487 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:36.381891 (+ 404us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:44:36.381994 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:44:36.382102 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:36.382253 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 02:44:36.382337 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:44:36.385821 12635 raft_consensus.cc:1261] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345 [term 1 FOLLOWER]: Refusing update from remote peer c088d4b02c704a7ba40c01da541a3fa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:36.385810 12777 raft_consensus.cc:1261] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931 [term 1 FOLLOWER]: Refusing update from remote peer c088d4b02c704a7ba40c01da541a3fa8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:36.386611 12947 consensus_queue.cc:1035] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d19a41448a394698b14bfc67ffd20931" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:36.386932 12942 consensus_queue.cc:1035] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feb3f982ef3a4a64a3ecf29fdea39345" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:36.389678 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.388969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.131:42399)
0925 02:44:36.389288 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.389435 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260925 02:44:36.392109 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.391488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.130:36873)
0925 02:44:36.391741 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.391870 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260925 02:44:36.394806 12951 mvcc.cc:205] Tried to move back new op lower bound from 7333086316069756928 to 7333086315812777984. Current Snapshot: MvccSnapshot[applied={T|T < 7333086316069756928}]
I20260925 02:44:36.395479 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.395065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.129:33737)
0925 02:44:36.395222 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.395311 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 02:44:36.396843 12945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.396179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.130:49197)
0925 02:44:36.396502 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.396627 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185}
I20260925 02:44:36.398310 12953 mvcc.cc:205] Tried to move back new op lower bound from 7333086316069756928 to 7333086315812777984. Current Snapshot: MvccSnapshot[applied={T|T < 7333086316069756928 or (T in {7333086316069756928})}]
I20260925 02:44:36.406279 12946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.405398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.129:45507)
0925 02:44:36.405964 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.406072 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448}
I20260925 02:44:36.411384 12944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.410810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.5.56.131:41289)
0925 02:44:36.411070 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.411189 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135}
I20260925 02:44:36.412147 12354 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:36.412506 12354 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52692:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:36.418884 12354 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fbfd3860e2ff46929abee5d1d5212edf: Sending DeleteTablet for 3 replicas of tablet ff381d4b14014cf5984c4d3866d65404
I20260925 02:44:36.419864 12615 tablet_service.cc:1514] Processing DeleteTablet for tablet ff381d4b14014cf5984c4d3866d65404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:36 UTC) from {username='oryx', principal='oryx/127.5.56.188@KRBTEST.COM'} at 127.0.0.1:56444
I20260925 02:44:36.420157 12757 tablet_service.cc:1514] Processing DeleteTablet for tablet ff381d4b14014cf5984c4d3866d65404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:36 UTC) from {username='oryx', principal='oryx/127.5.56.188@KRBTEST.COM'} at 127.0.0.1:41730
I20260925 02:44:36.420305 12468 tablet_service.cc:1514] Processing DeleteTablet for tablet ff381d4b14014cf5984c4d3866d65404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:36 UTC) from {username='oryx', principal='oryx/127.5.56.188@KRBTEST.COM'} at 127.0.0.1:36216
I20260925 02:44:36.420676 12756 tablet_service.cc:1514] Processing DeleteTablet for tablet ff381d4b14014cf5984c4d3866d65404 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:36 UTC) from {username='oryx', principal='oryx/127.5.56.188@KRBTEST.COM'} at 127.0.0.1:41730
I20260925 02:44:36.420981 12960 tablet_replica.cc:330] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931: stopping tablet replica
I20260925 02:44:36.421249 12960 raft_consensus.cc:2229] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:36.421650 12960 raft_consensus.cc:2258] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:36.422334 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 12326
I20260925 02:44:36.425175 12960 ts_tablet_manager.cc:1905] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:36.426842 12959 tablet_replica.cc:330] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345: stopping tablet replica
I20260925 02:44:36.427121 12959 raft_consensus.cc:2229] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:36.427502 12959 raft_consensus.cc:2258] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:36.431401 12960 ts_tablet_manager.cc:1918] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:36.431622 12960 log.cc:1196] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/ff381d4b14014cf5984c4d3866d65404
I20260925 02:44:36.432165 12960 ts_tablet_manager.cc:1939] T ff381d4b14014cf5984c4d3866d65404 P d19a41448a394698b14bfc67ffd20931: Deleting consensus metadata
I20260925 02:44:36.433647 12338 catalog_manager.cc:4841] TS d19a41448a394698b14bfc67ffd20931 (127.5.56.131:34367): tablet ff381d4b14014cf5984c4d3866d65404 (table test-table [id=26033ac981ce4d9a8aadc62ac768dbe5]) successfully deleted
W20260925 02:44:36.434224 12338 catalog_manager.cc:4808] TS d19a41448a394698b14bfc67ffd20931 (127.5.56.131:34367): delete failed for tablet ff381d4b14014cf5984c4d3866d65404 because the tablet was not found. No further retry: Not found: Tablet not found: ff381d4b14014cf5984c4d3866d65404
I20260925 02:44:36.434263 12961 tablet_replica.cc:330] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8: stopping tablet replica
I20260925 02:44:36.434509 12961 raft_consensus.cc:2229] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:44:36.434947 12961 raft_consensus.cc:2258] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:36.436004 12961 ts_tablet_manager.cc:1905] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:36.442835 12961 ts_tablet_manager.cc:1918] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:36.443065 12961 log.cc:1196] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/ff381d4b14014cf5984c4d3866d65404
I20260925 02:44:36.443579 12961 ts_tablet_manager.cc:1939] T ff381d4b14014cf5984c4d3866d65404 P c088d4b02c704a7ba40c01da541a3fa8: Deleting consensus metadata
I20260925 02:44:36.443994 12959 ts_tablet_manager.cc:1905] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:36.444729 12340 catalog_manager.cc:4841] TS c088d4b02c704a7ba40c01da541a3fa8 (127.5.56.129:46809): tablet ff381d4b14014cf5984c4d3866d65404 (table test-table [id=26033ac981ce4d9a8aadc62ac768dbe5]) successfully deleted
I20260925 02:44:36.448338 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.423358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36238 (local address 127.5.56.129:46809)
0925 02:44:36.423552 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:36.423558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:36.423779 (+ 221us) server_negotiation.cc:407] Connection header received
0925 02:44:36.424024 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:36.424032 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:36.424135 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:36.424265 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:36.426038 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.427602 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.429308 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.430240 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.438451 (+ 8211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:36.438697 (+ 246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:36.438707 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:36.438724 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:36.438747 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:36.441991 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:36.443045 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:36.443057 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:36.443059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:36.443257 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:36.446533 (+ 3276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:36.446539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:36.446543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:36.447062 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:36.447389 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:36.447751 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 02:44:36.447996 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:44:36.449266 12529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:36.449281 12959 ts_tablet_manager.cc:1918] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:36.449383 12959 log.cc:1196] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/ff381d4b14014cf5984c4d3866d65404
I20260925 02:44:36.449867 12959 ts_tablet_manager.cc:1939] T ff381d4b14014cf5984c4d3866d65404 P feb3f982ef3a4a64a3ecf29fdea39345: Deleting consensus metadata
I20260925 02:44:36.451567 12339 catalog_manager.cc:4841] TS feb3f982ef3a4a64a3ecf29fdea39345 (127.5.56.130:35741): tablet ff381d4b14014cf5984c4d3866d65404 (table test-table [id=26033ac981ce4d9a8aadc62ac768dbe5]) successfully deleted
I20260925 02:44:36.572312 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 12326
I20260925 02:44:36.595873 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 12546
I20260925 02:44:36.608927 12928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.596744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56452 (local address 127.5.56.130:35741)
0925 02:44:36.596932 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:36.596937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:36.596974 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:44:36.597133 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:36.597138 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:36.597200 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:36.597285 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:36.598635 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.599332 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.600623 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.601038 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.601130 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:36.603701 (+ 2571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:36.603715 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:36.603732 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:36.603765 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:36.606518 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:36.607196 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:36.607203 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:36.607205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:36.607343 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:36.607656 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:36.607659 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:36.607661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:36.608048 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:36.608267 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:36.608499 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 02:44:36.608669 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:44:36.609694 12676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:36.663385 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.662644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:36324)
0925 02:44:36.663114 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.663214 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":261,"threads_started":1}
I20260925 02:44:36.664642 12965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.664307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42051 (local address 127.0.0.1:36336)
0925 02:44:36.664458 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:36.664533 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:42051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260925 02:44:36.706502 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 12546
I20260925 02:44:36.725519 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 12694
I20260925 02:44:36.739908 12967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:36.726599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41734 (local address 127.5.56.131:34367)
0925 02:44:36.727059 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:36.727063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:36.727078 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:44:36.727133 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:36.727136 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:36.727196 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:36.727322 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:36.728699 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.729394 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.730721 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:36.731134 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:36.731273 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:36.733951 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:36.733969 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:36.733987 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:36.734020 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:36.736923 (+ 2903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:36.737612 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:36.737620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:36.737622 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:36.737788 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:36.738212 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:36.738219 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:36.738222 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:36.738720 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:36.739031 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:36.739317 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 02:44:36.739555 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":224,"threads_started":1}
I20260925 02:44:36.740638 12817 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:36.835963 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 12694
I20260925 02:44:36.854645 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 12021
I20260925 02:44:36.855799 12198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:36.977082 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 12021
I20260925 02:44:36.999213 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 12122
I20260925 02:44:37.000308 12385 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:37.125726 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 12122
2026-09-25T02:44:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12284 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:44:37 dist-test-slave-n563 krb5kdc[12973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:44:37 dist-test-slave-n563 krb5kdc[12973](info): set up 2 sockets
Sep 25 02:44:37 dist-test-slave-n563 krb5kdc[12973](info): commencing operation
krb5kdc: starting...
W20260925 02:44:38.244828 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.000s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:44:38 dist-test-slave-n563 krb5kdc[12973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:44:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:38Z Disabled control of system clock
WARNING: no policy specified for oryx/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.190.keytab.
Entry for principal oryx/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.190.keytab.
I20260925 02:44:38.475867 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43939
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.190.keytab
--principal=oryx/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:43939
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:38.810004 12986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:38.810302 12986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:38.810468 12986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:38.817730 12986 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:38.817844 12986 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 02:44:38.817885 12986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:38.817947 12986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:38.818001 12986 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:38.818056 12986 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:38.828282 12986 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:43939
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.190.keytab
--principal=oryx/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:38.829596 12986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:38.830966 12986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:38.843698 12992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:38.843652 12994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:38.843757 12991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:38.844485 12986 server_base.cc:1034] running on GCE node
I20260925 02:44:38.845263 12986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:38.846954 12986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:38.848220 12986 hybrid_clock.cc:648] HybridClock initialized: now 1790304278848155 us; error 46 us; skew 500 ppm
I20260925 02:44:38.848598 12986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:38 dist-test-slave-n563 krb5kdc[12973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304278, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:38.851902 12986 init.cc:377] Logged in from keytab as oryx/127.5.56.190@KRBTEST.COM (short username oryx)
I20260925 02:44:38.853423 12986 webserver.cc:460] Webserver started at http://127.5.56.190:33063/ using document root <none> and password file <none>
I20260925 02:44:38.854179 12986 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:38.854323 12986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:38.854696 12986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:38.857833 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "e6bcbcfc9ec14a31b5ecda5393efb8a0"
format_stamp: "Formatted at 2026-09-25 02:44:38 on dist-test-slave-n563"
server_key: "ca8cfd81ed0c20bb771d9dedd6a016a7"
server_key_iv: "07f4a67153ddf89b8fa0c665b0bd14c6"
server_key_version: "encryptionkey@0"
I20260925 02:44:38.858753 12986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "e6bcbcfc9ec14a31b5ecda5393efb8a0"
format_stamp: "Formatted at 2026-09-25 02:44:38 on dist-test-slave-n563"
server_key: "ca8cfd81ed0c20bb771d9dedd6a016a7"
server_key_iv: "07f4a67153ddf89b8fa0c665b0bd14c6"
server_key_version: "encryptionkey@0"
I20260925 02:44:38.864109 12986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:44:38.867811 13001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:38.869228 12986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:44:38.869423 12986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "e6bcbcfc9ec14a31b5ecda5393efb8a0"
format_stamp: "Formatted at 2026-09-25 02:44:38 on dist-test-slave-n563"
server_key: "ca8cfd81ed0c20bb771d9dedd6a016a7"
server_key_iv: "07f4a67153ddf89b8fa0c665b0bd14c6"
server_key_version: "encryptionkey@0"
I20260925 02:44:38.869674 12986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:38.890125 12986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:38.894222 12986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:38.894589 12986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:38.917423 12986 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:43939
I20260925 02:44:38.917443 13063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:43939 every 8 connection(s)
I20260925 02:44:38.918874 12986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:38.919307 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 12986
I20260925 02:44:38.919536 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:38.920435 5346 external_mini_cluster.cc:1428] Setting key e0a6d7abc7260a915d37b7c7fc8a3c8d
I20260925 02:44:38.923707 13064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:44:38 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304278, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:38.937382 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0: Bootstrap starting.
I20260925 02:44:38.941843 13064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:38.943280 13064 log.cc:824] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:38.945655 13067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:38.922564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37466 (local address 127.5.56.190:43939)
0925 02:44:38.923384 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:38.923396 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:38.923437 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:44:38.925106 (+ 1669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:38.925128 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:38.925518 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:38.925818 (+ 300us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:38.927618 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:38.928974 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:38.930909 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:38.931668 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:38.931873 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:38.936648 (+ 4775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:38.936663 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:38.936684 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:38.936733 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:38.940835 (+ 4102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:38.941655 (+ 820us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:38.941666 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:38.941676 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:38.941907 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:38.942363 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:38.942370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:38.942373 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:38.943153 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:38.943466 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:38.943937 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 02:44:38.944369 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":405,"threads_started":1}
I20260925 02:44:38.947093 13064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0: No bootstrap required, opened a new log
I20260925 02:44:38.954931 13064 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bcbcfc9ec14a31b5ecda5393efb8a0" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43939 } }
I20260925 02:44:38.955375 13064 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:38.955488 13064 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6bcbcfc9ec14a31b5ecda5393efb8a0, State: Initialized, Role: FOLLOWER
I20260925 02:44:38.956146 13064 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bcbcfc9ec14a31b5ecda5393efb8a0" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43939 } }
I20260925 02:44:38.956399 13064 raft_consensus.cc:388] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:38.956564 13064 raft_consensus.cc:482] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:38.956754 13064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:38.959414 13064 raft_consensus.cc:504] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bcbcfc9ec14a31b5ecda5393efb8a0" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43939 } }
I20260925 02:44:38.959970 13064 leader_election.cc:302] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e6bcbcfc9ec14a31b5ecda5393efb8a0; no voters:
I20260925 02:44:38.960585 13064 leader_election.cc:288] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:38.960724 13069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:38.961472 13069 raft_consensus.cc:686] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [term 1 LEADER]: Becoming Leader. State: Replica: e6bcbcfc9ec14a31b5ecda5393efb8a0, State: Running, Role: LEADER
I20260925 02:44:38.962071 13069 consensus_queue.cc:237] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bcbcfc9ec14a31b5ecda5393efb8a0" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43939 } }
I20260925 02:44:38.962925 13064 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:38.965020 13071 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e6bcbcfc9ec14a31b5ecda5393efb8a0. Latest consensus state: current_term: 1 leader_uuid: "e6bcbcfc9ec14a31b5ecda5393efb8a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bcbcfc9ec14a31b5ecda5393efb8a0" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43939 } } }
I20260925 02:44:38.965046 13070 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e6bcbcfc9ec14a31b5ecda5393efb8a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e6bcbcfc9ec14a31b5ecda5393efb8a0" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43939 } } }
I20260925 02:44:38.965548 13071 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:38.965641 13070 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:38.975095 13075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:38.979625 13075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:38.988792 13075 catalog_manager.cc:1347] Generated new cluster ID: 9986ecd2fdb64cb89c2f4c3507016a97
I20260925 02:44:38.988926 13075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:38.999217 13075 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:39.000599 13075 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:39.019764 13075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0: Generated new TSK 0
I20260925 02:44:39.020704 13075 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab.
Entry for principal oryx/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab.
I20260925 02:44:39.082991 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43939
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab
--principal=oryx/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:39.447173 13090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:39.447572 13090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:39.447774 13090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:39.455591 13090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:39.455734 13090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:39.455880 13090 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:44:39.456008 13090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:44:39.466835 13090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab
--principal=oryx/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.56.190:43939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:39.468360 13090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:39.469861 13090 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:39.484656 13098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:39.484735 13095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:39.484758 13096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:39.485646 13090 server_base.cc:1034] running on GCE node
I20260925 02:44:39.486300 13090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:39.487396 13090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:39.488632 13090 hybrid_clock.cc:648] HybridClock initialized: now 1790304279488558 us; error 42 us; skew 500 ppm
I20260925 02:44:39.488978 13090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:39 dist-test-slave-n563 krb5kdc[12973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304279, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:39.492494 13090 init.cc:377] Logged in from keytab as oryx/127.5.56.129@KRBTEST.COM (short username oryx)
I20260925 02:44:39.494154 13090 webserver.cc:460] Webserver started at http://127.5.56.129:42815/ using document root <none> and password file <none>
I20260925 02:44:39.494980 13090 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:39.495076 13090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:39.495352 13090 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:39.498759 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "e0e7ee78e54245adbbb730287ab408e6"
format_stamp: "Formatted at 2026-09-25 02:44:39 on dist-test-slave-n563"
server_key: "43b01cfebe048d87237e1972b45ab92b"
server_key_iv: "be5001212c4b124c38534def31eb936d"
server_key_version: "encryptionkey@0"
I20260925 02:44:39.499629 13090 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "e0e7ee78e54245adbbb730287ab408e6"
format_stamp: "Formatted at 2026-09-25 02:44:39 on dist-test-slave-n563"
server_key: "43b01cfebe048d87237e1972b45ab92b"
server_key_iv: "be5001212c4b124c38534def31eb936d"
server_key_version: "encryptionkey@0"
I20260925 02:44:39.505508 13090 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:44:39.509663 13105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:39.511374 13090 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:44:39.511602 13090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "e0e7ee78e54245adbbb730287ab408e6"
format_stamp: "Formatted at 2026-09-25 02:44:39 on dist-test-slave-n563"
server_key: "43b01cfebe048d87237e1972b45ab92b"
server_key_iv: "be5001212c4b124c38534def31eb936d"
server_key_version: "encryptionkey@0"
I20260925 02:44:39.511850 13090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:39.530054 13090 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:39.535167 13090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:39.535483 13090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:39.538578 13090 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:39.538703 13090 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:39.538885 13090 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:39.539000 13090 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:44:39 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304279, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:39.568250 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:39.539776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:47291 (local address 127.5.56.190:43939)
0925 02:44:39.540346 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:39.540356 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:39.541808 (+ 1452us) server_negotiation.cc:407] Connection header received
0925 02:44:39.544851 (+ 3043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:39.544858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:39.544981 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:39.545129 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:39.548529 (+ 3400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:39.549735 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:39.553429 (+ 3694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:39.554117 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:39.554313 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:39.559614 (+ 5301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:39.559620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:39.559624 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:39.559664 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:39.563473 (+ 3809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:39.564973 (+ 1500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:39.564979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:39.564981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:39.565151 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:39.565892 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:39.565897 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:39.565899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:39.566232 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:39.566467 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:39.567697 (+ 1230us) server_negotiation.cc:299] Negotiation successful
0925 02:44:39.567936 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":282,"threads_started":1}
I20260925 02:44:39.570634 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:39.540447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43939 (local address 127.5.56.129:47291)
0925 02:44:39.541554 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:39.541625 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:39.544453 (+ 2828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:39.545519 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:39.545539 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:39.546776 (+ 1237us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:39.548144 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:39.548185 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:39.550000 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:39.550010 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:39.553102 (+ 3092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:39.553125 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:39.554667 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:39.554676 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:39.554908 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:39.555818 (+ 910us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:39.555845 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:39.559193 (+ 3348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:39.563688 (+ 4495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:39.563705 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:39.563736 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:39.564737 (+ 1001us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:39.565292 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:39.565301 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:39.565307 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:39.565698 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:39.566416 (+ 718us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:39.566429 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:39.567109 (+ 680us) client_negotiation.cc:769] Sending connection context
0925 02:44:39.567538 (+ 429us) client_negotiation.cc:240] Negotiation successful
0925 02:44:39.568352 (+ 814us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"thread_start_us":489,"threads_started":1}
I20260925 02:44:39.596544 13114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:39.577971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:48895 (local address 127.5.56.190:43939)
0925 02:44:39.578224 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:39.578234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:39.579056 (+ 822us) server_negotiation.cc:407] Connection header received
0925 02:44:39.579248 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:39.579258 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:39.579381 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:39.579544 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:39.580717 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:39.582052 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:39.584478 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:39.585196 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:39.585381 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:39.588796 (+ 3415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:39.588802 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:39.588805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:39.588846 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:39.592504 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:39.594032 (+ 1528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:39.594040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:39.594044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:39.594257 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:39.594943 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:39.594947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:39.594949 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:39.595268 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:39.595418 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:39.596064 (+ 646us) server_negotiation.cc:299] Negotiation successful
0925 02:44:39.596270 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:44:39.597295 13175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:39.578056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43939 (local address 127.5.56.129:48895)
0925 02:44:39.578911 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:39.578934 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:39.579099 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:39.579601 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:39.579605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:39.580165 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:39.580472 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:39.580483 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:39.582248 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:39.582256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:39.584244 (+ 1988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:39.584268 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:39.585529 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:39.585542 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:39.585856 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:39.586800 (+ 944us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:39.586830 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:39.588465 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:39.592789 (+ 4324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:39.592801 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:39.592808 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:39.593789 (+ 981us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:39.594482 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:39.594488 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:39.594497 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:39.594721 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:39.595415 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:39.595423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:39.595725 (+ 302us) client_negotiation.cc:769] Sending connection context
0925 02:44:39.595923 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 02:44:39.596438 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":500,"threads_started":1}
I20260925 02:44:39.604485 13090 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:36527
I20260925 02:44:39.604512 13226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:36527 every 8 connection(s)
I20260925 02:44:39.605827 13090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:44:39.608224 13227 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43939
I20260925 02:44:39.608496 13227 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:39.609227 13227 heartbeater.cc:511] Master 127.5.56.190:43939 requested a full tablet report, sending...
I20260925 02:44:39.611666 13019 ts_manager.cc:194] Registered new tserver with Master: e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129:36527)
I20260925 02:44:39.611693 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 13090
I20260925 02:44:39.611948 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:44:39.612751 5346 external_mini_cluster.cc:1428] Setting key 699a36d4942ea7ad095433589e709301
I20260925 02:44:39.613336 13019 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:47291
WARNING: no policy specified for oryx/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab.
Entry for principal oryx/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab.
I20260925 02:44:39.666738 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43939
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab
--principal=oryx/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:40.030818 13231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:40.031209 13231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:40.031394 13231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:40.039049 13231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:40.039247 13231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:40.039419 13231 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:44:40.039573 13231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:44:40.050148 13231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab
--principal=oryx/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.56.190:43939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:40.051680 13231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:40.053170 13231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:40.067770 13237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:40.067668 13236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:40.067742 13239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:40.068677 13231 server_base.cc:1034] running on GCE node
I20260925 02:44:40.069430 13231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:40.070570 13231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:40.071851 13231 hybrid_clock.cc:648] HybridClock initialized: now 1790304280071749 us; error 62 us; skew 500 ppm
I20260925 02:44:40.072259 13231 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:40 dist-test-slave-n563 krb5kdc[12973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304280, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:40.076193 13231 init.cc:377] Logged in from keytab as oryx/127.5.56.130@KRBTEST.COM (short username oryx)
I20260925 02:44:40.078035 13231 webserver.cc:460] Webserver started at http://127.5.56.130:44333/ using document root <none> and password file <none>
I20260925 02:44:40.078928 13231 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:40.079073 13231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:40.079473 13231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:40.083310 13231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "adbd13e8f1d6436492a3646e7d68a792"
format_stamp: "Formatted at 2026-09-25 02:44:40 on dist-test-slave-n563"
server_key: "a4ab4e8b13cbe4cc8ddb780fbdc023c8"
server_key_iv: "10026372253fd8f990ff37b10b0018ae"
server_key_version: "encryptionkey@0"
I20260925 02:44:40.084158 13231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "adbd13e8f1d6436492a3646e7d68a792"
format_stamp: "Formatted at 2026-09-25 02:44:40 on dist-test-slave-n563"
server_key: "a4ab4e8b13cbe4cc8ddb780fbdc023c8"
server_key_iv: "10026372253fd8f990ff37b10b0018ae"
server_key_version: "encryptionkey@0"
I20260925 02:44:40.090041 13231 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260925 02:44:40.094106 13246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:40.095836 13231 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:44:40.096092 13231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "adbd13e8f1d6436492a3646e7d68a792"
format_stamp: "Formatted at 2026-09-25 02:44:40 on dist-test-slave-n563"
server_key: "a4ab4e8b13cbe4cc8ddb780fbdc023c8"
server_key_iv: "10026372253fd8f990ff37b10b0018ae"
server_key_version: "encryptionkey@0"
I20260925 02:44:40.096328 13231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:40.122570 13231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:40.127604 13231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:40.127928 13231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:40.130889 13231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:40.131042 13231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:40.131198 13231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:40.131265 13231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:44:40 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304280, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:40.160017 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.131812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:38819 (local address 127.5.56.190:43939)
0925 02:44:40.132308 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:40.132313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:40.133706 (+ 1393us) server_negotiation.cc:407] Connection header received
0925 02:44:40.135977 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:40.135985 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:40.136083 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:40.136223 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:40.138952 (+ 2729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.140146 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.143743 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.144353 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.144543 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.151047 (+ 6504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:40.151055 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:40.151062 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:40.151117 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:40.154262 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:40.155679 (+ 1417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:40.155688 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:40.155694 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:40.155890 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:40.156622 (+ 732us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:40.156630 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:40.156635 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:40.157086 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:40.157276 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:40.159016 (+ 1740us) server_negotiation.cc:299] Negotiation successful
0925 02:44:40.159412 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":262,"threads_started":1}
I20260925 02:44:40.160579 13256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.132596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43939 (local address 127.5.56.130:38819)
0925 02:44:40.133521 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:40.133563 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:40.135655 (+ 2092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:40.136547 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:40.136562 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:40.137509 (+ 947us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:40.138707 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.138759 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.140383 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.140390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.143497 (+ 3107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.143524 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.144708 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.144719 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.145125 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.146237 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:40.146282 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:40.150602 (+ 4320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:40.154516 (+ 3914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:40.154533 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:40.154568 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:40.155470 (+ 902us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:40.156044 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:40.156049 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:40.156053 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:40.156425 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:40.157369 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:40.157377 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:40.158005 (+ 628us) client_negotiation.cc:769] Sending connection context
0925 02:44:40.158467 (+ 462us) client_negotiation.cc:240] Negotiation successful
0925 02:44:40.159178 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":417,"threads_started":1}
I20260925 02:44:40.187758 13255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.171421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:43735 (local address 127.5.56.190:43939)
0925 02:44:40.171675 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:40.171686 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:40.172509 (+ 823us) server_negotiation.cc:407] Connection header received
0925 02:44:40.172789 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:40.172794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:40.172926 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:40.173075 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:40.174490 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.175291 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.176807 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.177256 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.177413 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.180609 (+ 3196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:40.180619 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:40.180627 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:40.180684 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:40.184243 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:40.185175 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:40.185181 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:40.185185 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:40.185369 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:40.185879 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:40.185885 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:40.185889 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:40.186510 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:40.186699 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:40.187140 (+ 441us) server_negotiation.cc:299] Negotiation successful
0925 02:44:40.187386 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:44:40.187875 13328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.171494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43939 (local address 127.5.56.130:43735)
0925 02:44:40.172346 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:40.172373 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:40.172589 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:40.173103 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:40.173109 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:40.173836 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:40.174263 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.174275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.175464 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.175473 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.176613 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.176626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.177508 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.177519 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.177714 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.178546 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:40.178582 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:40.180175 (+ 1593us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:40.184416 (+ 4241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:40.184422 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:40.184427 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:40.184924 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:40.185545 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:40.185556 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:40.185561 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:40.185741 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:40.186640 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:40.186646 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:40.186857 (+ 211us) client_negotiation.cc:769] Sending connection context
0925 02:44:40.187034 (+ 177us) client_negotiation.cc:240] Negotiation successful
0925 02:44:40.187403 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":632,"thread_start_us":421,"threads_started":1}
I20260925 02:44:40.195566 13231 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:45629
I20260925 02:44:40.195590 13367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:45629 every 8 connection(s)
I20260925 02:44:40.196890 13231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:44:40.199136 13368 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43939
I20260925 02:44:40.199398 13368 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:40.200157 13368 heartbeater.cc:511] Master 127.5.56.190:43939 requested a full tablet report, sending...
I20260925 02:44:40.201738 13018 ts_manager.cc:194] Registered new tserver with Master: adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629)
I20260925 02:44:40.202760 13018 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:38819
I20260925 02:44:40.203670 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 13231
I20260925 02:44:40.203869 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:44:40.204337 5346 external_mini_cluster.cc:1428] Setting key 8e8164a139e1cee6a7f1522597ea09e2
WARNING: no policy specified for oryx/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab.
Entry for principal oryx/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab.
I20260925 02:44:40.255293 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43939
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab
--principal=oryx/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:40.603404 13372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:40.603734 13372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:40.603915 13372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:40.611724 13372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:40.611861 13372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:40.612057 13372 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 02:44:40.612206 13372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:44:40.616676 13227 heartbeater.cc:503] Master 127.5.56.190:43939 was elected leader, sending a full tablet report...
I20260925 02:44:40.623452 13372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab
--principal=oryx/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.5.56.190:43939
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:40.624908 13372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:40.626362 13372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:40.640827 13377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:40.640753 13380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:40.640789 13378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:40.641992 13372 server_base.cc:1034] running on GCE node
I20260925 02:44:40.642838 13372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:40.643843 13372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:40.645080 13372 hybrid_clock.cc:648] HybridClock initialized: now 1790304280645002 us; error 50 us; skew 500 ppm
I20260925 02:44:40.645429 13372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:40 dist-test-slave-n563 krb5kdc[12973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304280, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:40.648828 13372 init.cc:377] Logged in from keytab as oryx/127.5.56.131@KRBTEST.COM (short username oryx)
I20260925 02:44:40.650660 13372 webserver.cc:460] Webserver started at http://127.5.56.131:40521/ using document root <none> and password file <none>
I20260925 02:44:40.651446 13372 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:40.651597 13372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:40.651960 13372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:40.655292 13372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "17c4b515cf264194ac6f229571f10055"
format_stamp: "Formatted at 2026-09-25 02:44:40 on dist-test-slave-n563"
server_key: "3f958d90355e8e8680c67ca3c542dd2c"
server_key_iv: "a25faabb9bbdf6b43b84045ec9985e82"
server_key_version: "encryptionkey@0"
I20260925 02:44:40.656188 13372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "17c4b515cf264194ac6f229571f10055"
format_stamp: "Formatted at 2026-09-25 02:44:40 on dist-test-slave-n563"
server_key: "3f958d90355e8e8680c67ca3c542dd2c"
server_key_iv: "a25faabb9bbdf6b43b84045ec9985e82"
server_key_version: "encryptionkey@0"
I20260925 02:44:40.662030 13372 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:44:40.666368 13387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:40.668107 13372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:44:40.668313 13372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "17c4b515cf264194ac6f229571f10055"
format_stamp: "Formatted at 2026-09-25 02:44:40 on dist-test-slave-n563"
server_key: "3f958d90355e8e8680c67ca3c542dd2c"
server_key_iv: "a25faabb9bbdf6b43b84045ec9985e82"
server_key_version: "encryptionkey@0"
I20260925 02:44:40.668587 13372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:40.686559 13372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:40.691640 13372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:40.691980 13372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:40.695099 13372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:40.695225 13372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:40.695410 13372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:40.695489 13372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:44:40 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304280, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:40.722955 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.695843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:40125 (local address 127.5.56.190:43939)
0925 02:44:40.696502 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:40.696509 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:40.697839 (+ 1330us) server_negotiation.cc:407] Connection header received
0925 02:44:40.700516 (+ 2677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:40.700525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:40.700639 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:40.700727 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:40.703781 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.705235 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.707972 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.708843 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.709047 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.715296 (+ 6249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:40.715305 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:40.715309 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:40.715366 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:40.719170 (+ 3804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:40.720267 (+ 1097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:40.720273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:40.720277 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:40.720433 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:40.720973 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:40.720978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:40.720980 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:40.721272 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:40.721454 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:40.722267 (+ 813us) server_negotiation.cc:299] Negotiation successful
0925 02:44:40.722541 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":397,"threads_started":1}
I20260925 02:44:40.724108 13397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.696601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43939 (local address 127.5.56.131:40125)
0925 02:44:40.697628 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:40.697680 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:40.700070 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:40.701509 (+ 1439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:40.701518 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:40.702474 (+ 956us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:40.703487 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.703515 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.705474 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.705481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.707703 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.707726 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.709251 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.709263 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.709732 (+ 469us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.710731 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:40.710769 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:40.714906 (+ 4137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:40.719377 (+ 4471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:40.719387 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:40.719437 (+ 50us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:40.720116 (+ 679us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:40.720555 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:40.720561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:40.720564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:40.720791 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:40.721430 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:40.721440 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:40.721860 (+ 420us) client_negotiation.cc:769] Sending connection context
0925 02:44:40.722154 (+ 294us) client_negotiation.cc:240] Negotiation successful
0925 02:44:40.722820 (+ 666us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":496,"threads_started":1}
I20260925 02:44:40.748131 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.730854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43939 (local address 127.5.56.131:40057)
0925 02:44:40.731605 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:40.731629 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:40.731835 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:40.732276 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:40.732282 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:40.732810 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:40.733161 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.733172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.734306 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.734316 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.736696 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.736723 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.737630 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.737639 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.737942 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.738830 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:40.738864 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:40.740327 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:40.744601 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:40.744605 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:40.744608 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:40.745070 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:40.745557 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:40.745564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:40.745569 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:40.745770 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:40.746570 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:40.746575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:40.746925 (+ 350us) client_negotiation.cc:769] Sending connection context
0925 02:44:40.747124 (+ 199us) client_negotiation.cc:240] Negotiation successful
0925 02:44:40.747600 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":427,"threads_started":1}
I20260925 02:44:40.748955 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.730783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:40057 (local address 127.5.56.190:43939)
0925 02:44:40.731007 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:40.731012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:40.731751 (+ 739us) server_negotiation.cc:407] Connection header received
0925 02:44:40.731998 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:40.732005 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:40.732102 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:40.732221 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:40.733340 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.734092 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.736919 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.737385 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.737496 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.740585 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:40.740591 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:40.740594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:40.740635 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:40.744457 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:40.745210 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:40.745219 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:40.745225 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:40.745401 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:40.745947 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:40.745951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:40.745954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:40.746417 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:40.746595 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:40.748486 (+ 1891us) server_negotiation.cc:299] Negotiation successful
0925 02:44:40.748715 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260925 02:44:40.758466 13372 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:45313
I20260925 02:44:40.758639 13508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:45313 every 8 connection(s)
I20260925 02:44:40.759873 13372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:44:40.760496 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 13372
I20260925 02:44:40.760685 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:44:40.761273 5346 external_mini_cluster.cc:1428] Setting key 15bfa7ba1f74a4acaaec5689ef68f706
I20260925 02:44:40.762768 13509 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43939
I20260925 02:44:40.763073 13509 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:40.763990 13509 heartbeater.cc:511] Master 127.5.56.190:43939 requested a full tablet report, sending...
I20260925 02:44:40.765487 13019 ts_manager.cc:194] Registered new tserver with Master: 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:40.766345 13019 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:40125
I20260925 02:44:40.777549 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 02:44:40 dist-test-slave-n563 krb5kdc[12973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 02:44:40 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:40 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:44:40.832355 13396 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:40.820846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37480 (local address 127.5.56.190:43939)
0925 02:44:40.821084 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:40.821091 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:40.821286 (+ 195us) server_negotiation.cc:407] Connection header received
0925 02:44:40.821540 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:40.821545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:40.821613 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:40.821720 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:40.823253 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.824000 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.825319 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.825841 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.825991 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.831993 (+ 6002us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37480: BlockingRecv error: recv got EOF from 127.0.0.1:37480 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 25 02:44:40 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304280, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:40.850953 13396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.835937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37488 (local address 127.5.56.190:43939)
0925 02:44:40.836224 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:40.836237 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:40.836334 (+ 97us) server_negotiation.cc:407] Connection header received
0925 02:44:40.836498 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:40.836502 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:40.836570 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:40.836658 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:40.837994 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.838902 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.840245 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.840732 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.840871 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.845343 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:40.845349 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:40.845352 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:40.845397 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:40.848115 (+ 2718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:40.848878 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:40.848882 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:40.848885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:40.849011 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:40.849348 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:40.849352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:40.849354 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:40.849645 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:40.849846 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:40.850291 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 02:44:40.850616 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260925 02:44:40.855643 13019 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:44:40.863083 13019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:44:40.891927 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.877316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35256 (local address 127.5.56.129:36527)
0925 02:44:40.878030 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:40.878045 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:40.878671 (+ 626us) server_negotiation.cc:407] Connection header received
0925 02:44:40.879813 (+ 1142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:40.879827 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:40.880205 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:40.880441 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:40.881712 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.883930 (+ 2218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.888179 (+ 4249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.890066 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.890163 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.890338 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:40.891350 (+ 1012us) server_negotiation.cc:299] Negotiation successful
0925 02:44:40.891566 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":296,"threads_started":1}
I20260925 02:44:40.893360 13528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.877231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36527 (local address 127.0.0.1:35256)
0925 02:44:40.878490 (+ 1259us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:40.878555 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:40.879149 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:40.880834 (+ 1685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:40.880843 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:40.880889 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:40.881414 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.881460 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.884255 (+ 2795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.884264 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.887871 (+ 3607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.887900 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.890622 (+ 2722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.890631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.890969 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.891046 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 02:44:40.891254 (+ 208us) client_negotiation.cc:240] Negotiation successful
0925 02:44:40.892912 (+ 1658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":894,"mutex_wait_us":1415,"thread_start_us":1039,"threads_started":1}
I20260925 02:44:40.897596 13529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.877308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:45629 (local address 127.0.0.1:57130)
0925 02:44:40.879401 (+ 2093us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:40.879431 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:40.879557 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:40.890831 (+ 11274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:40.890837 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:40.890868 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:40.891710 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.891728 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.893406 (+ 1678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.893414 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.895084 (+ 1670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.895097 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.896455 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.896462 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.896988 (+ 526us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.897034 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:44:40.897131 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:44:40.897290 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1470,"mutex_wait_us":76,"thread_start_us":702,"threads_started":1}
I20260925 02:44:40.898250 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.877859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57130 (local address 127.5.56.130:45629)
0925 02:44:40.889889 (+ 12030us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:40.889904 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:40.889948 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:44:40.890088 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:40.890102 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:40.890545 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:40.890975 (+ 430us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:40.891942 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.893261 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.895257 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.896230 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.896392 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.896656 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:40.897885 (+ 1229us) server_negotiation.cc:299] Negotiation successful
0925 02:44:40.898051 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11775,"thread_start_us":11789,"threads_started":1}
I20260925 02:44:40.897769 13157 tablet_service.cc:1467] Processing CreateTablet for tablet 99073447fcef4c0188859a613a41a90e (DEFAULT_TABLE table=test-table [id=74c0492eadc74b3584f269c9847ec8c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:40.900614 13531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.879194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45313 (local address 127.0.0.1:39158)
0925 02:44:40.889378 (+ 10184us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:40.889406 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:40.889568 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:40.892029 (+ 2461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:40.892034 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:40.892062 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:40.892466 (+ 404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.892481 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.894568 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.894574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.896710 (+ 2136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:40.896725 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.899815 (+ 3090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.899820 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:40.899990 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.900017 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 02:44:40.900090 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:44:40.900196 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10026,"mutex_wait_us":4,"thread_start_us":9873,"threads_started":1}
I20260925 02:44:40.900826 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:40.879750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39158 (local address 127.5.56.131:45313)
0925 02:44:40.891039 (+ 11289us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:40.891048 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:40.891083 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:44:40.891184 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:40.891195 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:40.891459 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:40.891675 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:40.892725 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.894359 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.896924 (+ 2565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:40.899593 (+ 2669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:40.899735 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:40.900146 (+ 411us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:40.900227 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 02:44:40.900508 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11103,"thread_start_us":10895,"threads_started":1}
I20260925 02:44:40.902456 13297 tablet_service.cc:1467] Processing CreateTablet for tablet 99073447fcef4c0188859a613a41a90e (DEFAULT_TABLE table=test-table [id=74c0492eadc74b3584f269c9847ec8c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:40.903218 13157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99073447fcef4c0188859a613a41a90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:40.903754 13297 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99073447fcef4c0188859a613a41a90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:40.904647 13438 tablet_service.cc:1467] Processing CreateTablet for tablet 99073447fcef4c0188859a613a41a90e (DEFAULT_TABLE table=test-table [id=74c0492eadc74b3584f269c9847ec8c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:40.906196 13438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99073447fcef4c0188859a613a41a90e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:40.915287 13533 tablet_bootstrap.cc:492] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792: Bootstrap starting.
I20260925 02:44:40.922178 13535 tablet_bootstrap.cc:492] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: Bootstrap starting.
I20260925 02:44:40.923516 13534 tablet_bootstrap.cc:492] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: Bootstrap starting.
I20260925 02:44:40.926548 13535 tablet_bootstrap.cc:654] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:40.926748 13533 tablet_bootstrap.cc:654] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:40.927206 13534 tablet_bootstrap.cc:654] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:40.928187 13533 log.cc:824] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:40.928419 13534 log.cc:824] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:40.929279 13535 log.cc:824] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:40.941928 13533 tablet_bootstrap.cc:492] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792: No bootstrap required, opened a new log
I20260925 02:44:40.941929 13534 tablet_bootstrap.cc:492] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: No bootstrap required, opened a new log
I20260925 02:44:40.942271 13534 ts_tablet_manager.cc:1397] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.008s
I20260925 02:44:40.942281 13533 ts_tablet_manager.cc:1397] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.012s
I20260925 02:44:40.942447 13535 tablet_bootstrap.cc:492] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: No bootstrap required, opened a new log
I20260925 02:44:40.942739 13535 ts_tablet_manager.cc:1397] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.011s
I20260925 02:44:40.950054 13534 raft_consensus.cc:348] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:40.950572 13534 raft_consensus.cc:374] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:40.950678 13534 raft_consensus.cc:729] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e7ee78e54245adbbb730287ab408e6, State: Initialized, Role: FOLLOWER
I20260925 02:44:40.951588 13534 consensus_queue.cc:260] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:40.953979 13535 raft_consensus.cc:348] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:40.953979 13533 raft_consensus.cc:348] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:40.954490 13533 raft_consensus.cc:374] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:40.954485 13535 raft_consensus.cc:374] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:40.954594 13535 raft_consensus.cc:729] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c4b515cf264194ac6f229571f10055, State: Initialized, Role: FOLLOWER
I20260925 02:44:40.954593 13533 raft_consensus.cc:729] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbd13e8f1d6436492a3646e7d68a792, State: Initialized, Role: FOLLOWER
I20260925 02:44:40.954613 13534 ts_tablet_manager.cc:1428] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: Time spent starting tablet: real 0.012s user 0.002s sys 0.009s
I20260925 02:44:40.955348 13533 consensus_queue.cc:260] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:40.955348 13535 consensus_queue.cc:260] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:40.956545 13509 heartbeater.cc:503] Master 127.5.56.190:43939 was elected leader, sending a full tablet report...
I20260925 02:44:40.957082 13368 heartbeater.cc:503] Master 127.5.56.190:43939 was elected leader, sending a full tablet report...
I20260925 02:44:40.957294 13535 ts_tablet_manager.cc:1428] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260925 02:44:40.958678 13533 ts_tablet_manager.cc:1428] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
W20260925 02:44:41.011914 13510 tablet.cc:2369] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:44:41.063035 13540 raft_consensus.cc:482] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:44:41.063562 13540 raft_consensus.cc:504] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.064934 13539 raft_consensus.cc:482] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:44:41.065256 13539 raft_consensus.cc:504] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.065806 13540 leader_election.cc:288] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129:36527), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:41.076309 13530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.068307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:42955 (local address 127.5.56.130:45629)
0925 02:44:41.068551 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:41.068560 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:41.068587 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:44:41.068669 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:41.068678 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:41.068810 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:41.068943 (+ 133us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:41.069752 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.070980 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.073700 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.074981 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.075117 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.075189 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:41.075862 (+ 673us) server_negotiation.cc:299] Negotiation successful
0925 02:44:41.076002 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 02:44:41.076299 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.066429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:42705 (local address 127.5.56.129:36527)
0925 02:44:41.066686 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:41.066696 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:41.067838 (+ 1142us) server_negotiation.cc:407] Connection header received
0925 02:44:41.068128 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:41.068138 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:41.068256 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:41.068352 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:41.069229 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.070612 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.073649 (+ 3037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.074969 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.075111 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.075189 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:41.075855 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 02:44:41.076002 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":69}
I20260925 02:44:41.077138 13542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.066174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36527 (local address 127.5.56.130:42705)
0925 02:44:41.067710 (+ 1536us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:41.067726 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:41.067909 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:41.068591 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:41.068596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:41.068627 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:41.069035 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.069051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.070834 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.070840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.073432 (+ 2592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.073452 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.075476 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.075482 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.075662 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.075693 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:44:41.076607 (+ 914us) client_negotiation.cc:240] Negotiation successful
0925 02:44:41.076762 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1400,"thread_start_us":1171,"threads_started":1}
I20260925 02:44:41.078306 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.067687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:45629 (local address 127.5.56.129:42955)
0925 02:44:41.068408 (+ 721us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:41.068426 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:41.068529 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:41.068984 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:41.068992 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:41.069024 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:41.069568 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.069583 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.071167 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.071173 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.073514 (+ 2341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.073535 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.075473 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.075479 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.075659 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.075693 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:44:41.076928 (+ 1235us) client_negotiation.cc:240] Negotiation successful
0925 02:44:41.077081 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":26,"thread_start_us":1411,"threads_started":1}
I20260925 02:44:41.078481 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99073447fcef4c0188859a613a41a90e" candidate_uuid: "adbd13e8f1d6436492a3646e7d68a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e7ee78e54245adbbb730287ab408e6" is_pre_election: true
I20260925 02:44:41.079089 13182 raft_consensus.cc:2454] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adbd13e8f1d6436492a3646e7d68a792 in term 0.
I20260925 02:44:41.080358 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99073447fcef4c0188859a613a41a90e" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbd13e8f1d6436492a3646e7d68a792" is_pre_election: true
I20260925 02:44:41.080931 13322 raft_consensus.cc:2454] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 0.
I20260925 02:44:41.081279 13539 leader_election.cc:288] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:41.082134 13249 leader_election.cc:302] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adbd13e8f1d6436492a3646e7d68a792, e0e7ee78e54245adbbb730287ab408e6; no voters:
I20260925 02:44:41.082692 13540 raft_consensus.cc:2781] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:44:41.082856 13540 raft_consensus.cc:482] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:44:41.082954 13540 raft_consensus.cc:3037] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:41.085336 13106 leader_election.cc:302] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adbd13e8f1d6436492a3646e7d68a792, e0e7ee78e54245adbbb730287ab408e6; no voters:
I20260925 02:44:41.085834 13539 raft_consensus.cc:2781] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:44:41.086007 13539 raft_consensus.cc:482] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:44:41.086143 13539 raft_consensus.cc:3037] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:41.086472 13540 raft_consensus.cc:504] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.087868 13532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.066891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:39301 (local address 127.5.56.131:45313)
0925 02:44:41.067137 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:41.067145 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:41.076911 (+ 9766us) server_negotiation.cc:407] Connection header received
0925 02:44:41.077537 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:41.077543 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:41.077653 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:41.077741 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:41.078720 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.080136 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.084441 (+ 4305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.086206 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.086416 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.086530 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:41.087437 (+ 907us) server_negotiation.cc:299] Negotiation successful
0925 02:44:41.087587 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":80}
I20260925 02:44:41.087846 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99073447fcef4c0188859a613a41a90e" candidate_uuid: "adbd13e8f1d6436492a3646e7d68a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e7ee78e54245adbbb730287ab408e6"
I20260925 02:44:41.088483 13544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.070198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45313 (local address 127.5.56.129:34729)
0925 02:44:41.078589 (+ 8391us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:41.078615 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:41.078743 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:41.079447 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:41.079452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:41.079479 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:41.080036 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.080049 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.081697 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.081703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.083666 (+ 1963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.083701 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.087564 (+ 3863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.087572 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.087813 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.087884 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 02:44:41.087990 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:44:41.088164 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8283,"thread_start_us":16731,"threads_started":1}
I20260925 02:44:41.088641 13540 leader_election.cc:288] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [CANDIDATE]: Term 1 election: Requested vote from peers e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129:36527), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:41.088944 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.066324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45313 (local address 127.5.56.130:39301)
0925 02:44:41.076768 (+ 10444us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:41.076800 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:41.077320 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:41.078020 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:41.078025 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:41.078061 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:44:41.078514 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.078536 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.080709 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.080714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.084161 (+ 3447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.084187 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.087007 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.087015 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.087209 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.087245 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:44:41.088212 (+ 967us) client_negotiation.cc:240] Negotiation successful
0925 02:44:41.088395 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10251,"thread_start_us":10616,"threads_started":1}
I20260925 02:44:41.089388 13546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.070851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:34729 (local address 127.5.56.131:45313)
0925 02:44:41.077897 (+ 7046us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:41.077902 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:41.078865 (+ 963us) server_negotiation.cc:407] Connection header received
0925 02:44:41.078935 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:41.078939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:41.079004 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:41.079106 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:44:41.080257 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.081522 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.083911 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.086638 (+ 2727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.087380 (+ 742us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.087456 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:41.088865 (+ 1409us) server_negotiation.cc:299] Negotiation successful
0925 02:44:41.089022 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1103,"server-negotiator.queue_time_us":6868,"thread_start_us":6591,"threads_started":1}
I20260925 02:44:41.090179 13463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99073447fcef4c0188859a613a41a90e" candidate_uuid: "adbd13e8f1d6436492a3646e7d68a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055"
I20260925 02:44:41.090240 13464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99073447fcef4c0188859a613a41a90e" candidate_uuid: "adbd13e8f1d6436492a3646e7d68a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055" is_pre_election: true
I20260925 02:44:41.090266 13462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99073447fcef4c0188859a613a41a90e" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055" is_pre_election: true
I20260925 02:44:41.090996 13539 raft_consensus.cc:504] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.091933 13539 leader_election.cc:288] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 election: Requested vote from peers adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:41.093152 13464 raft_consensus.cc:2454] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adbd13e8f1d6436492a3646e7d68a792 in term 0.
I20260925 02:44:41.093637 13182 raft_consensus.cc:2379] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate adbd13e8f1d6436492a3646e7d68a792 in current term 1: Already voted for candidate e0e7ee78e54245adbbb730287ab408e6 in this term.
I20260925 02:44:41.093513 13461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99073447fcef4c0188859a613a41a90e" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055"
I20260925 02:44:41.093791 13461 raft_consensus.cc:3037] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:41.093945 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99073447fcef4c0188859a613a41a90e" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbd13e8f1d6436492a3646e7d68a792"
I20260925 02:44:41.094264 13322 raft_consensus.cc:2379] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e0e7ee78e54245adbbb730287ab408e6 in current term 1: Already voted for candidate adbd13e8f1d6436492a3646e7d68a792 in this term.
I20260925 02:44:41.095372 13249 leader_election.cc:302] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: adbd13e8f1d6436492a3646e7d68a792; no voters: 17c4b515cf264194ac6f229571f10055, e0e7ee78e54245adbbb730287ab408e6
I20260925 02:44:41.095692 13540 raft_consensus.cc:2726] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 02:44:41.096918 13461 raft_consensus.cc:2454] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 1.
I20260925 02:44:41.097344 13109 leader_election.cc:302] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 17c4b515cf264194ac6f229571f10055, e0e7ee78e54245adbbb730287ab408e6; no voters: adbd13e8f1d6436492a3646e7d68a792
I20260925 02:44:41.097666 13539 raft_consensus.cc:2781] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:41.098106 13539 raft_consensus.cc:686] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 1 LEADER]: Becoming Leader. State: Replica: e0e7ee78e54245adbbb730287ab408e6, State: Running, Role: LEADER
I20260925 02:44:41.098657 13539 consensus_queue.cc:237] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.103433 13019 catalog_manager.cc:5495] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129). New cstate: current_term: 1 leader_uuid: "e0e7ee78e54245adbbb730287ab408e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } health_report { overall_health: UNKNOWN } } }
W20260925 02:44:41.108743 13228 tablet.cc:2369] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:44:41.140830 13527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.134869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35266 (local address 127.5.56.129:36527)
0925 02:44:41.135102 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:41.135109 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:41.135245 (+ 136us) server_negotiation.cc:407] Connection header received
0925 02:44:41.135535 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:41.135539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:41.135654 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:41.135828 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:44:41.136207 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.137008 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.138420 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.138888 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.139039 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.139507 (+ 468us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:44:41.140131 (+ 624us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:44:41.140304 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:41.140388 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:44:41.140485 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 02:44:41.150091 13461 raft_consensus.cc:1261] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Refusing update from remote peer e0e7ee78e54245adbbb730287ab408e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:41.150118 13322 raft_consensus.cc:1261] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Refusing update from remote peer e0e7ee78e54245adbbb730287ab408e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:41.151108 13547 consensus_queue.cc:1035] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:41.151535 13539 consensus_queue.cc:1035] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:41.164754 13550 mvcc.cc:205] Tried to move back new op lower bound from 7333086335579291648 to 7333086335383932928. Current Snapshot: MvccSnapshot[applied={T|T < 7333086335579291648}]
I20260925 02:44:41.181730 13018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:41.182060 13018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:41.185928 13018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0: Sending DeleteTablet for 3 replicas of tablet 99073447fcef4c0188859a613a41a90e
I20260925 02:44:41.187352 13297 tablet_service.cc:1514] Processing DeleteTablet for tablet 99073447fcef4c0188859a613a41a90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:41 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:57130
I20260925 02:44:41.187350 13157 tablet_service.cc:1514] Processing DeleteTablet for tablet 99073447fcef4c0188859a613a41a90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:41 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:35256
I20260925 02:44:41.187697 13438 tablet_service.cc:1514] Processing DeleteTablet for tablet 99073447fcef4c0188859a613a41a90e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:41 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:39158
I20260925 02:44:41.193427 13018 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37488:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:44:41.195254 13018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:44:41.198997 13562 tablet_replica.cc:330] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: stopping tablet replica
I20260925 02:44:41.199680 13562 raft_consensus.cc:2229] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:41.200840 13562 raft_consensus.cc:2258] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:41.204187 13562 ts_tablet_manager.cc:1905] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:41.206967 13296 tablet_service.cc:1467] Processing CreateTablet for tablet a1292904795842d1be901c362cf2b418 (DEFAULT_TABLE table=test-table [id=03afff02d171422eaa8b4de3007d151a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:41.207669 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1292904795842d1be901c362cf2b418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:41.211858 13156 tablet_service.cc:1467] Processing CreateTablet for tablet a1292904795842d1be901c362cf2b418 (DEFAULT_TABLE table=test-table [id=03afff02d171422eaa8b4de3007d151a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:41.212598 13156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1292904795842d1be901c362cf2b418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:41.217546 13560 tablet_replica.cc:330] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792: stopping tablet replica
I20260925 02:44:41.218091 13560 raft_consensus.cc:2229] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:41.220057 13562 ts_tablet_manager.cc:1918] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:41.220286 13562 log.cc:1196] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/99073447fcef4c0188859a613a41a90e
I20260925 02:44:41.220847 13562 ts_tablet_manager.cc:1939] T 99073447fcef4c0188859a613a41a90e P 17c4b515cf264194ac6f229571f10055: Deleting consensus metadata
I20260925 02:44:41.220731 13438 tablet_service.cc:1467] Processing CreateTablet for tablet a1292904795842d1be901c362cf2b418 (DEFAULT_TABLE table=test-table [id=03afff02d171422eaa8b4de3007d151a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:41.223120 13005 catalog_manager.cc:4841] TS 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313): tablet 99073447fcef4c0188859a613a41a90e (table test-table [id=74c0492eadc74b3584f269c9847ec8c5]) successfully deleted
I20260925 02:44:41.223248 13560 raft_consensus.cc:2258] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:41.223663 13438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1292904795842d1be901c362cf2b418. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:41.224296 13561 tablet_replica.cc:330] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: stopping tablet replica
I20260925 02:44:41.224844 13561 raft_consensus.cc:2229] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:44:41.225610 13561 raft_consensus.cc:2258] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:41.225687 13560 ts_tablet_manager.cc:1905] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:41.227726 13561 ts_tablet_manager.cc:1905] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:41.237926 13561 ts_tablet_manager.cc:1918] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:41.238204 13561 log.cc:1196] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/99073447fcef4c0188859a613a41a90e
I20260925 02:44:41.238894 13561 ts_tablet_manager.cc:1939] T 99073447fcef4c0188859a613a41a90e P e0e7ee78e54245adbbb730287ab408e6: Deleting consensus metadata
I20260925 02:44:41.241078 13004 catalog_manager.cc:4841] TS e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129:36527): tablet 99073447fcef4c0188859a613a41a90e (table test-table [id=74c0492eadc74b3584f269c9847ec8c5]) successfully deleted
I20260925 02:44:41.241317 13560 ts_tablet_manager.cc:1918] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:41.241506 13560 log.cc:1196] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/99073447fcef4c0188859a613a41a90e
I20260925 02:44:41.242101 13560 ts_tablet_manager.cc:1939] T 99073447fcef4c0188859a613a41a90e P adbd13e8f1d6436492a3646e7d68a792: Deleting consensus metadata
I20260925 02:44:41.242317 13533 tablet_bootstrap.cc:492] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792: Bootstrap starting.
I20260925 02:44:41.244199 13533 tablet_bootstrap.cc:654] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:41.244354 13002 catalog_manager.cc:4841] TS adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629): tablet 99073447fcef4c0188859a613a41a90e (table test-table [id=74c0492eadc74b3584f269c9847ec8c5]) successfully deleted
I20260925 02:44:41.245779 13534 tablet_bootstrap.cc:492] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6: Bootstrap starting.
I20260925 02:44:41.248440 13534 tablet_bootstrap.cc:654] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:41.251703 13535 tablet_bootstrap.cc:492] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055: Bootstrap starting.
I20260925 02:44:41.254065 13535 tablet_bootstrap.cc:654] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:41.258663 13533 tablet_bootstrap.cc:492] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792: No bootstrap required, opened a new log
I20260925 02:44:41.258935 13533 ts_tablet_manager.cc:1397] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.000s
I20260925 02:44:41.259869 13533 raft_consensus.cc:348] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.260250 13533 raft_consensus.cc:374] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:41.260412 13533 raft_consensus.cc:729] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbd13e8f1d6436492a3646e7d68a792, State: Initialized, Role: FOLLOWER
I20260925 02:44:41.260742 13533 consensus_queue.cc:260] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.261649 13533 ts_tablet_manager.cc:1428] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:41.263476 13535 tablet_bootstrap.cc:492] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055: No bootstrap required, opened a new log
I20260925 02:44:41.263674 13535 ts_tablet_manager.cc:1397] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20260925 02:44:41.265053 13534 tablet_bootstrap.cc:492] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6: No bootstrap required, opened a new log
I20260925 02:44:41.265244 13534 ts_tablet_manager.cc:1397] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.003s
I20260925 02:44:41.265264 13535 raft_consensus.cc:348] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.265566 13535 raft_consensus.cc:374] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:41.265671 13535 raft_consensus.cc:729] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c4b515cf264194ac6f229571f10055, State: Initialized, Role: FOLLOWER
I20260925 02:44:41.266000 13535 consensus_queue.cc:260] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.266662 13535 ts_tablet_manager.cc:1428] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260925 02:44:41.267654 13534 raft_consensus.cc:348] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.267905 13534 raft_consensus.cc:374] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:41.268023 13534 raft_consensus.cc:729] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e7ee78e54245adbbb730287ab408e6, State: Initialized, Role: FOLLOWER
I20260925 02:44:41.268350 13534 consensus_queue.cc:260] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.269104 13534 ts_tablet_manager.cc:1428] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
W20260925 02:44:41.450877 13369 tablet.cc:2369] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:44:41.555395 13540 raft_consensus.cc:482] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:44:41.555673 13540 raft_consensus.cc:504] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.556517 13540 leader_election.cc:288] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129:36527), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:41.557157 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1292904795842d1be901c362cf2b418" candidate_uuid: "adbd13e8f1d6436492a3646e7d68a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e7ee78e54245adbbb730287ab408e6" is_pre_election: true
I20260925 02:44:41.557153 13461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1292904795842d1be901c362cf2b418" candidate_uuid: "adbd13e8f1d6436492a3646e7d68a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055" is_pre_election: true
I20260925 02:44:41.557451 13182 raft_consensus.cc:2454] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adbd13e8f1d6436492a3646e7d68a792 in term 0.
I20260925 02:44:41.557444 13461 raft_consensus.cc:2454] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adbd13e8f1d6436492a3646e7d68a792 in term 0.
I20260925 02:44:41.557979 13250 leader_election.cc:302] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c4b515cf264194ac6f229571f10055, adbd13e8f1d6436492a3646e7d68a792; no voters:
I20260925 02:44:41.558341 13540 raft_consensus.cc:2781] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:44:41.558487 13540 raft_consensus.cc:482] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:44:41.558609 13540 raft_consensus.cc:3037] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:41.561717 13540 raft_consensus.cc:504] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.562565 13540 leader_election.cc:288] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [CANDIDATE]: Term 1 election: Requested vote from peers e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129:36527), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:41.563225 13182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1292904795842d1be901c362cf2b418" candidate_uuid: "adbd13e8f1d6436492a3646e7d68a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0e7ee78e54245adbbb730287ab408e6"
I20260925 02:44:41.563227 13461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1292904795842d1be901c362cf2b418" candidate_uuid: "adbd13e8f1d6436492a3646e7d68a792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055"
I20260925 02:44:41.563464 13461 raft_consensus.cc:3037] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:41.563472 13182 raft_consensus.cc:3037] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:41.566027 13182 raft_consensus.cc:2454] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adbd13e8f1d6436492a3646e7d68a792 in term 1.
I20260925 02:44:41.566659 13249 leader_election.cc:302] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adbd13e8f1d6436492a3646e7d68a792, e0e7ee78e54245adbbb730287ab408e6; no voters:
I20260925 02:44:41.566748 13461 raft_consensus.cc:2454] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adbd13e8f1d6436492a3646e7d68a792 in term 1.
I20260925 02:44:41.566987 13540 raft_consensus.cc:2781] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:41.567564 13540 raft_consensus.cc:686] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 1 LEADER]: Becoming Leader. State: Replica: adbd13e8f1d6436492a3646e7d68a792, State: Running, Role: LEADER
I20260925 02:44:41.567951 13540 consensus_queue.cc:237] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.572161 13018 catalog_manager.cc:5495] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 reported cstate change: term changed from 0 to 1, leader changed from <none> to adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130). New cstate: current_term: 1 leader_uuid: "adbd13e8f1d6436492a3646e7d68a792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } health_report { overall_health: UNKNOWN } } }
Sep 25 02:44:41 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304278, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.190@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:41.672901 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.657021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43939 (local address 127.0.0.1:37492)
0925 02:44:41.657926 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:41.657943 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:41.658096 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:41.658482 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:41.658486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:41.659204 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:41.659423 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.659432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.660531 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.660535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.661511 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.661524 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.662294 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.662299 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.662517 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.663253 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:41.663272 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:41.665948 (+ 2676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:41.669709 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:41.669769 (+ 60us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:41.669779 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:41.670479 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:41.670975 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:41.670978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:41.670981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:41.671129 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:41.671765 (+ 636us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:41.671773 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:41.671953 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 02:44:41.672136 (+ 183us) client_negotiation.cc:240] Negotiation successful
0925 02:44:41.672462 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"thread_start_us":542,"threads_started":1}
I20260925 02:44:41.673368 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.657032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37492 (local address 127.5.56.190:43939)
0925 02:44:41.657881 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:41.657887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:41.658045 (+ 158us) server_negotiation.cc:407] Connection header received
0925 02:44:41.658208 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:41.658211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:41.658275 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:41.658374 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:41.659582 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.660376 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.661684 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.662120 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.662230 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.666207 (+ 3977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:41.666213 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:41.666216 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:41.666250 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:41.669498 (+ 3248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:41.670623 (+ 1125us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:41.670627 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:41.670629 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:41.670858 (+ 229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:41.671249 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:41.671253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:41.671254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:41.671616 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:41.671736 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:41.672269 (+ 533us) server_negotiation.cc:299] Negotiation successful
0925 02:44:41.672721 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":136,"server-negotiator.queue_time_us":659,"thread_start_us":361,"threads_started":1}
I20260925 02:44:41.679201 13018 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:37492:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 02:44:41.680435 13018 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:44:41.687970 13156 tablet_service.cc:1467] Processing CreateTablet for tablet 8619b6521f5b4eff82ecc5116e414532 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fc5841625e248babfbaa6955e8dc8dd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:44:41.688026 13296 tablet_service.cc:1467] Processing CreateTablet for tablet 8619b6521f5b4eff82ecc5116e414532 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fc5841625e248babfbaa6955e8dc8dd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:44:41.688580 13156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8619b6521f5b4eff82ecc5116e414532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:41.688386 13438 tablet_service.cc:1467] Processing CreateTablet for tablet 8619b6521f5b4eff82ecc5116e414532 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7fc5841625e248babfbaa6955e8dc8dd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 02:44:41.688611 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8619b6521f5b4eff82ecc5116e414532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:41.688932 13438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8619b6521f5b4eff82ecc5116e414532. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:41.694798 13534 tablet_bootstrap.cc:492] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: Bootstrap starting.
I20260925 02:44:41.696200 13533 tablet_bootstrap.cc:492] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792: Bootstrap starting.
I20260925 02:44:41.696413 13534 tablet_bootstrap.cc:654] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:41.696789 13535 tablet_bootstrap.cc:492] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055: Bootstrap starting.
I20260925 02:44:41.698014 13533 tablet_bootstrap.cc:654] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:41.698559 13535 tablet_bootstrap.cc:654] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:41.699144 13534 tablet_bootstrap.cc:492] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: No bootstrap required, opened a new log
I20260925 02:44:41.699329 13534 ts_tablet_manager.cc:1397] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260925 02:44:41.700083 13534 raft_consensus.cc:348] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.700379 13534 raft_consensus.cc:374] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:41.700474 13534 raft_consensus.cc:729] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e7ee78e54245adbbb730287ab408e6, State: Initialized, Role: FOLLOWER
I20260925 02:44:41.700546 13533 tablet_bootstrap.cc:492] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792: No bootstrap required, opened a new log
I20260925 02:44:41.700716 13533 ts_tablet_manager.cc:1397] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:44:41.700779 13534 consensus_queue.cc:260] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.701508 13533 raft_consensus.cc:348] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.701427 13539 tablet_replica.cc:439] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } } }
I20260925 02:44:41.701713 13534 ts_tablet_manager.cc:1428] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:44:41.701766 13533 raft_consensus.cc:374] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:41.701820 13539 tablet_replica.cc:442] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:44:41.701843 13533 raft_consensus.cc:729] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbd13e8f1d6436492a3646e7d68a792, State: Initialized, Role: FOLLOWER
I20260925 02:44:41.702099 13533 consensus_queue.cc:260] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.702760 13533 ts_tablet_manager.cc:1428] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 02:44:41.702677 13540 tablet_replica.cc:439] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } } }
I20260925 02:44:41.703032 13540 tablet_replica.cc:442] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:44:41.703833 13535 tablet_bootstrap.cc:492] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055: No bootstrap required, opened a new log
I20260925 02:44:41.704026 13535 ts_tablet_manager.cc:1397] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260925 02:44:41.704794 13535 raft_consensus.cc:348] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.705106 13535 raft_consensus.cc:374] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:41.705193 13535 raft_consensus.cc:729] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c4b515cf264194ac6f229571f10055, State: Initialized, Role: FOLLOWER
I20260925 02:44:41.705449 13535 consensus_queue.cc:260] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.705953 13535 ts_tablet_manager.cc:1428] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260925 02:44:41.705945 13541 tablet_replica.cc:439] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } } }
I20260925 02:44:41.706367 13541 tablet_replica.cc:442] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:44:41.834571 13539 raft_consensus.cc:482] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:44:41.834828 13539 raft_consensus.cc:504] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.835582 13539 leader_election.cc:288] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:41.836172 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8619b6521f5b4eff82ecc5116e414532" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbd13e8f1d6436492a3646e7d68a792" is_pre_election: true
I20260925 02:44:41.836192 13461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8619b6521f5b4eff82ecc5116e414532" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055" is_pre_election: true
I20260925 02:44:41.836450 13461 raft_consensus.cc:2454] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 0.
I20260925 02:44:41.836448 13322 raft_consensus.cc:2454] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 0.
I20260925 02:44:41.836961 13106 leader_election.cc:302] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adbd13e8f1d6436492a3646e7d68a792, e0e7ee78e54245adbbb730287ab408e6; no voters:
I20260925 02:44:41.837277 13539 raft_consensus.cc:2781] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:44:41.837433 13539 raft_consensus.cc:482] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:44:41.837551 13539 raft_consensus.cc:3037] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:41.839715 13539 raft_consensus.cc:504] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.840344 13539 leader_election.cc:288] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 election: Requested vote from peers adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:41.841048 13461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8619b6521f5b4eff82ecc5116e414532" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055"
I20260925 02:44:41.841045 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8619b6521f5b4eff82ecc5116e414532" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbd13e8f1d6436492a3646e7d68a792"
I20260925 02:44:41.841325 13461 raft_consensus.cc:3037] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:41.841336 13322 raft_consensus.cc:3037] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:41.844468 13461 raft_consensus.cc:2454] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 1.
I20260925 02:44:41.844468 13322 raft_consensus.cc:2454] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 1.
I20260925 02:44:41.845054 13106 leader_election.cc:302] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17c4b515cf264194ac6f229571f10055, adbd13e8f1d6436492a3646e7d68a792, e0e7ee78e54245adbbb730287ab408e6; no voters:
I20260925 02:44:41.845531 13539 raft_consensus.cc:2781] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:41.845993 13539 raft_consensus.cc:686] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [term 1 LEADER]: Becoming Leader. State: Replica: e0e7ee78e54245adbbb730287ab408e6, State: Running, Role: LEADER
I20260925 02:44:41.846609 13539 consensus_queue.cc:237] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:41.848271 13578 tablet_replica.cc:439] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: TxnStatusTablet state changed. Reason: New leader e0e7ee78e54245adbbb730287ab408e6. Latest consensus state: current_term: 1 leader_uuid: "e0e7ee78e54245adbbb730287ab408e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } } }
I20260925 02:44:41.848582 13578 tablet_replica.cc:442] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:44:41.849221 13580 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 02:44:41.849474 13580 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 02:44:41.849926 13019 catalog_manager.cc:5495] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129). New cstate: current_term: 1 leader_uuid: "e0e7ee78e54245adbbb730287ab408e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } health_report { overall_health: UNKNOWN } } }
I20260925 02:44:41.898551 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.892639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36527 (local address 127.0.0.1:35276)
0925 02:44:41.892922 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:41.892942 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:41.893140 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:41.893731 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:41.893736 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:41.893751 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:44:41.893940 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.893950 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.895261 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.895267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.896305 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.896318 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.897127 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.897134 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.897308 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.897378 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:44:41.897833 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:44:41.897880 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 02:44:41.897977 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:44:41.898160 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 02:44:41.898603 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.892686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.5.56.129:36527)
0925 02:44:41.893402 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:41.893410 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:41.893429 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:44:41.893490 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:41.893493 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:41.893573 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:41.893694 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:44:41.894120 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.895113 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.896467 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.896948 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.897097 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.897606 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:44:41.897727 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:44:41.897831 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:41.898026 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 02:44:41.898115 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":381,"threads_started":1}
I20260925 02:44:41.903594 13322 raft_consensus.cc:1261] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Refusing update from remote peer e0e7ee78e54245adbbb730287ab408e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:41.903679 13461 raft_consensus.cc:1261] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Refusing update from remote peer e0e7ee78e54245adbbb730287ab408e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:41.904327 13578 consensus_queue.cc:1035] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:41.904894 13539 consensus_queue.cc:1035] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:41.911499 13540 tablet_replica.cc:439] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792: TxnStatusTablet state changed. Reason: New leader e0e7ee78e54245adbbb730287ab408e6. Latest consensus state: current_term: 1 leader_uuid: "e0e7ee78e54245adbbb730287ab408e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } } }
I20260925 02:44:41.911820 13540 tablet_replica.cc:442] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:44:41.917310 13539 tablet_replica.cc:439] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0e7ee78e54245adbbb730287ab408e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } } }
I20260925 02:44:41.917641 13539 tablet_replica.cc:442] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:44:41.925079 13540 tablet_replica.cc:439] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0e7ee78e54245adbbb730287ab408e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } } }
I20260925 02:44:41.925387 13540 tablet_replica.cc:442] T 8619b6521f5b4eff82ecc5116e414532 P adbd13e8f1d6436492a3646e7d68a792: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:44:41.936141 13541 tablet_replica.cc:439] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055: TxnStatusTablet state changed. Reason: New leader e0e7ee78e54245adbbb730287ab408e6. Latest consensus state: current_term: 1 leader_uuid: "e0e7ee78e54245adbbb730287ab408e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } } }
I20260925 02:44:41.936540 13541 tablet_replica.cc:442] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:44:41.938897 13539 tablet_replica.cc:439] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e0e7ee78e54245adbbb730287ab408e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } } }
I20260925 02:44:41.939265 13539 tablet_replica.cc:442] T 8619b6521f5b4eff82ecc5116e414532 P e0e7ee78e54245adbbb730287ab408e6: This TxnStatusTablet replica's current role is: LEADER
I20260925 02:44:41.945406 13585 mvcc.cc:205] Tried to move back new op lower bound from 7333086338670219264 to 7333086338445996032. Current Snapshot: MvccSnapshot[applied={T|T < 7333086338670219264}]
I20260925 02:44:41.948311 13541 tablet_replica.cc:439] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e0e7ee78e54245adbbb730287ab408e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } } }
I20260925 02:44:41.948668 13541 tablet_replica.cc:442] T 8619b6521f5b4eff82ecc5116e414532 P 17c4b515cf264194ac6f229571f10055: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 02:44:41.956826 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.934037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57138 (local address 127.5.56.130:45629)
0925 02:44:41.948890 (+ 14853us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:41.948897 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:41.948919 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:44:41.949068 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:41.949076 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:41.949213 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:41.950309 (+ 1096us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:44:41.950521 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.952049 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.954229 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.954750 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.954906 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.955436 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:44:41.956134 (+ 698us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:44:41.956304 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:41.956403 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:44:41.956503 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14698,"thread_start_us":15381,"threads_started":1}
I20260925 02:44:41.975450 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.968649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:44667 (local address 127.5.56.129:36527)
0925 02:44:41.968861 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:41.968867 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:41.969445 (+ 578us) server_negotiation.cc:407] Connection header received
0925 02:44:41.969639 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:41.969645 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:41.969747 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:41.969865 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:44:41.970364 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.971605 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.973291 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.973950 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.974100 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.974662 (+ 562us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:44:41.974775 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:44:41.974898 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:41.975035 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 02:44:41.975124 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:44:41.975544 13597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:41.968607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36527 (local address 127.5.56.130:44667)
0925 02:44:41.969336 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:41.969355 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:41.969489 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:41.969931 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:41.969934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:41.969944 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:44:41.970212 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.970220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.971790 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.971795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.973137 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:41.973150 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:41.974134 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:41.974141 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:41.974281 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:41.974345 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:44:41.974892 (+ 547us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:44:41.974904 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 02:44:41.975009 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 02:44:41.975156 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":400,"threads_started":1}
I20260925 02:44:41.988024 13182 raft_consensus.cc:1261] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Refusing update from remote peer adbd13e8f1d6436492a3646e7d68a792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:41.988865 13566 consensus_queue.cc:1035] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:41.988726 13461 raft_consensus.cc:1261] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Refusing update from remote peer adbd13e8f1d6436492a3646e7d68a792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:41.990167 13566 consensus_queue.cc:1035] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:41.996901 13586 mvcc.cc:205] Tried to move back new op lower bound from 7333086339014193152 to 7333086337306484736. Current Snapshot: MvccSnapshot[applied={T|T < 7333086339014193152}]
I20260925 02:44:42.034162 13593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:42.027342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:46691 (local address 127.5.56.130:45629)
0925 02:44:42.027528 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:42.027536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:42.028368 (+ 832us) server_negotiation.cc:407] Connection header received
0925 02:44:42.028632 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:42.028638 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:42.028729 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:42.028826 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:44:42.029579 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.030355 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.032071 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.032766 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.032899 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:42.033469 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:44:42.033579 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:44:42.033712 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:42.033841 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 02:44:42.033912 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:44:42.034499 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:42.027378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:45629 (local address 127.5.56.129:46691)
0925 02:44:42.028258 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:42.028277 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:42.028453 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:42.028865 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:42.028870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:42.028885 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:44:42.029313 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:42.029338 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.030523 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.030532 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:42.031920 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:42.031936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.032945 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.032951 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:42.033184 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:42.033267 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:44:42.033715 (+ 448us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:44:42.033729 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 02:44:42.033818 (+ 89us) client_negotiation.cc:240] Negotiation successful
0925 02:44:42.033980 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":431,"threads_started":1}
I20260925 02:44:42.079006 13019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:42.079310 13019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37488:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:42.083407 13019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0: Sending DeleteTablet for 3 replicas of tablet a1292904795842d1be901c362cf2b418
I20260925 02:44:42.084822 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet a1292904795842d1be901c362cf2b418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:42 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:57130
I20260925 02:44:42.085961 13438 tablet_service.cc:1514] Processing DeleteTablet for tablet a1292904795842d1be901c362cf2b418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:42 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:39158
I20260925 02:44:42.090644 13156 tablet_service.cc:1514] Processing DeleteTablet for tablet a1292904795842d1be901c362cf2b418 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:42 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:35256
I20260925 02:44:42.102807 13608 tablet_replica.cc:330] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055: stopping tablet replica
I20260925 02:44:42.103083 13608 raft_consensus.cc:2229] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:42.103382 13608 raft_consensus.cc:2258] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:42.104947 13609 tablet_replica.cc:330] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6: stopping tablet replica
I20260925 02:44:42.105188 13609 raft_consensus.cc:2229] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:42.105496 13609 raft_consensus.cc:2258] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:42.106915 13608 ts_tablet_manager.cc:1905] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:42.108045 13607 tablet_replica.cc:330] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792: stopping tablet replica
I20260925 02:44:42.114871 13607 raft_consensus.cc:2229] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:44:42.115780 13607 raft_consensus.cc:2258] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:42.115849 13609 ts_tablet_manager.cc:1905] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:42.117137 13607 ts_tablet_manager.cc:1905] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:42.123982 13609 ts_tablet_manager.cc:1918] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:44:42.124161 13609 log.cc:1196] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/a1292904795842d1be901c362cf2b418
I20260925 02:44:42.124675 13609 ts_tablet_manager.cc:1939] T a1292904795842d1be901c362cf2b418 P e0e7ee78e54245adbbb730287ab408e6: Deleting consensus metadata
I20260925 02:44:42.125998 13608 ts_tablet_manager.cc:1918] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:44:42.126163 13608 log.cc:1196] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/a1292904795842d1be901c362cf2b418
I20260925 02:44:42.126791 13608 ts_tablet_manager.cc:1939] T a1292904795842d1be901c362cf2b418 P 17c4b515cf264194ac6f229571f10055: Deleting consensus metadata
I20260925 02:44:42.128219 13005 catalog_manager.cc:4841] TS 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313): tablet a1292904795842d1be901c362cf2b418 (table test-table [id=03afff02d171422eaa8b4de3007d151a]) successfully deleted
I20260925 02:44:42.128986 13004 catalog_manager.cc:4841] TS e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129:36527): tablet a1292904795842d1be901c362cf2b418 (table test-table [id=03afff02d171422eaa8b4de3007d151a]) successfully deleted
I20260925 02:44:42.131009 13607 ts_tablet_manager.cc:1918] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:44:42.131175 13607 log.cc:1196] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/a1292904795842d1be901c362cf2b418
I20260925 02:44:42.131682 13607 ts_tablet_manager.cc:1939] T a1292904795842d1be901c362cf2b418 P adbd13e8f1d6436492a3646e7d68a792: Deleting consensus metadata
I20260925 02:44:42.134788 13002 catalog_manager.cc:4841] TS adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629): tablet a1292904795842d1be901c362cf2b418 (table test-table [id=03afff02d171422eaa8b4de3007d151a]) successfully deleted
Sep 25 02:44:42 dist-test-slave-n563 krb5kdc[12973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 02:44:42 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:42 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:44:42.188792 13573 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:42.169593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37508 (local address 127.5.56.190:43939)
0925 02:44:42.169816 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:42.169824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:42.175292 (+ 5468us) server_negotiation.cc:407] Connection header received
0925 02:44:42.175500 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:42.175507 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:42.175612 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:42.175740 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:42.177624 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.178870 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.180987 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.181703 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.181878 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:42.188499 (+ 6621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37508: BlockingRecv error: recv got EOF from 127.0.0.1:37508 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 25 02:44:42 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM
I20260925 02:44:42.213771 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:42.192933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37514 (local address 127.5.56.190:43939)
0925 02:44:42.193129 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:42.193137 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:42.193703 (+ 566us) server_negotiation.cc:407] Connection header received
0925 02:44:42.193891 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:42.193898 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:42.193995 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:42.194106 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:42.195761 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.196910 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.199287 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.199994 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.200113 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:42.206111 (+ 5998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:42.206119 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:42.206124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:42.206179 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:42.209982 (+ 3803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:42.211057 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:42.211063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:42.211066 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:42.211284 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:42.211838 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:42.211844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:42.211847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:42.212244 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:42.212520 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:42.213125 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 02:44:42.213387 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:44:42.227136 13019 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:44:42.228014 13019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:44:42.239099 13156 tablet_service.cc:1467] Processing CreateTablet for tablet bcb0096bab4f466288218cb4e283b82a (DEFAULT_TABLE table=test-table [id=e89f593e0e0f46c78b3e17db6aeea608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:42.239662 13156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb0096bab4f466288218cb4e283b82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:42.240388 13296 tablet_service.cc:1467] Processing CreateTablet for tablet bcb0096bab4f466288218cb4e283b82a (DEFAULT_TABLE table=test-table [id=e89f593e0e0f46c78b3e17db6aeea608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:42.240846 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb0096bab4f466288218cb4e283b82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:42.246975 13438 tablet_service.cc:1467] Processing CreateTablet for tablet bcb0096bab4f466288218cb4e283b82a (DEFAULT_TABLE table=test-table [id=e89f593e0e0f46c78b3e17db6aeea608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:42.247475 13438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bcb0096bab4f466288218cb4e283b82a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:42.275583 13624 tablet_bootstrap.cc:492] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6: Bootstrap starting.
I20260925 02:44:42.277833 13624 tablet_bootstrap.cc:654] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:42.281019 13623 tablet_bootstrap.cc:492] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055: Bootstrap starting.
I20260925 02:44:42.285261 13623 tablet_bootstrap.cc:654] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:42.290202 13622 tablet_bootstrap.cc:492] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792: Bootstrap starting.
I20260925 02:44:42.292505 13622 tablet_bootstrap.cc:654] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:42.297242 13623 tablet_bootstrap.cc:492] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055: No bootstrap required, opened a new log
I20260925 02:44:42.297501 13623 ts_tablet_manager.cc:1397] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20260925 02:44:42.298431 13623 raft_consensus.cc:348] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.298822 13623 raft_consensus.cc:374] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:42.298962 13623 raft_consensus.cc:729] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c4b515cf264194ac6f229571f10055, State: Initialized, Role: FOLLOWER
I20260925 02:44:42.299285 13623 consensus_queue.cc:260] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.301033 13624 tablet_bootstrap.cc:492] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6: No bootstrap required, opened a new log
I20260925 02:44:42.301254 13624 ts_tablet_manager.cc:1397] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6: Time spent bootstrapping tablet: real 0.026s user 0.008s sys 0.000s
I20260925 02:44:42.302295 13624 raft_consensus.cc:348] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.302645 13624 raft_consensus.cc:374] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:42.302680 13622 tablet_bootstrap.cc:492] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792: No bootstrap required, opened a new log
I20260925 02:44:42.302826 13624 raft_consensus.cc:729] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e7ee78e54245adbbb730287ab408e6, State: Initialized, Role: FOLLOWER
I20260925 02:44:42.302871 13622 ts_tablet_manager.cc:1397] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260925 02:44:42.303175 13624 consensus_queue.cc:260] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.303870 13624 ts_tablet_manager.cc:1428] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 02:44:42.303822 13622 raft_consensus.cc:348] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.304093 13622 raft_consensus.cc:374] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:42.304628 13623 ts_tablet_manager.cc:1428] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260925 02:44:42.305215 13622 raft_consensus.cc:729] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbd13e8f1d6436492a3646e7d68a792, State: Initialized, Role: FOLLOWER
I20260925 02:44:42.305580 13622 consensus_queue.cc:260] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.307864 13622 ts_tablet_manager.cc:1428] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260925 02:44:42.329794 13539 raft_consensus.cc:482] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:44:42.330053 13539 raft_consensus.cc:504] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.330933 13539 leader_election.cc:288] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:42.331689 13462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb0096bab4f466288218cb4e283b82a" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055" is_pre_election: true
I20260925 02:44:42.331691 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb0096bab4f466288218cb4e283b82a" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbd13e8f1d6436492a3646e7d68a792" is_pre_election: true
I20260925 02:44:42.331977 13462 raft_consensus.cc:2454] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 0.
I20260925 02:44:42.332001 13322 raft_consensus.cc:2454] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 0.
I20260925 02:44:42.332532 13106 leader_election.cc:302] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adbd13e8f1d6436492a3646e7d68a792, e0e7ee78e54245adbbb730287ab408e6; no voters:
I20260925 02:44:42.332821 13539 raft_consensus.cc:2781] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:44:42.332974 13539 raft_consensus.cc:482] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:44:42.333066 13539 raft_consensus.cc:3037] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:42.336323 13539 raft_consensus.cc:504] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.337025 13539 leader_election.cc:288] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 election: Requested vote from peers adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:42.337657 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb0096bab4f466288218cb4e283b82a" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbd13e8f1d6436492a3646e7d68a792"
I20260925 02:44:42.337905 13322 raft_consensus.cc:3037] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:42.338037 13462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bcb0096bab4f466288218cb4e283b82a" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055"
I20260925 02:44:42.338270 13462 raft_consensus.cc:3037] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:42.341326 13322 raft_consensus.cc:2454] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 1.
I20260925 02:44:42.341354 13462 raft_consensus.cc:2454] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 1.
I20260925 02:44:42.341974 13109 leader_election.cc:302] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c4b515cf264194ac6f229571f10055, e0e7ee78e54245adbbb730287ab408e6; no voters:
I20260925 02:44:42.342351 13539 raft_consensus.cc:2781] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:42.342602 13539 raft_consensus.cc:686] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 1 LEADER]: Becoming Leader. State: Replica: e0e7ee78e54245adbbb730287ab408e6, State: Running, Role: LEADER
I20260925 02:44:42.343000 13539 consensus_queue.cc:237] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.346273 13018 catalog_manager.cc:5495] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129). New cstate: current_term: 1 leader_uuid: "e0e7ee78e54245adbbb730287ab408e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } health_report { overall_health: UNKNOWN } } }
I20260925 02:44:42.375809 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:42.369514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35278 (local address 127.5.56.129:36527)
0925 02:44:42.369742 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:42.369748 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:42.369943 (+ 195us) server_negotiation.cc:407] Connection header received
0925 02:44:42.370197 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:42.370201 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:42.370289 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:42.370389 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:44:42.370811 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.371869 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.373783 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.374439 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.374592 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:42.375033 (+ 441us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:44:42.375150 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:44:42.375255 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:42.375386 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 02:44:42.375471 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 02:44:42.378999 13462 raft_consensus.cc:1261] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Refusing update from remote peer e0e7ee78e54245adbbb730287ab408e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:42.379815 13322 raft_consensus.cc:1261] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Refusing update from remote peer e0e7ee78e54245adbbb730287ab408e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:42.379776 13605 consensus_queue.cc:1035] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:42.380455 13605 consensus_queue.cc:1035] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:42.395388 13017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:42.395665 13017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:42.398882 13017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0: Sending DeleteTablet for 3 replicas of tablet bcb0096bab4f466288218cb4e283b82a
I20260925 02:44:42.399785 13156 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb0096bab4f466288218cb4e283b82a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:42 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:35256
I20260925 02:44:42.399796 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb0096bab4f466288218cb4e283b82a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:42 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:57130
I20260925 02:44:42.400105 13607 tablet_replica.cc:330] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792: stopping tablet replica
I20260925 02:44:42.400097 13609 tablet_replica.cc:330] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6: stopping tablet replica
I20260925 02:44:42.400309 13609 raft_consensus.cc:2229] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:44:42.400321 13607 raft_consensus.cc:2229] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:42.400558 13607 raft_consensus.cc:2258] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:42.400524 13438 tablet_service.cc:1514] Processing DeleteTablet for tablet bcb0096bab4f466288218cb4e283b82a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:42 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:39158
I20260925 02:44:42.400653 13609 raft_consensus.cc:2258] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:42.400833 13608 tablet_replica.cc:330] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055: stopping tablet replica
I20260925 02:44:42.401082 13608 raft_consensus.cc:2229] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:42.401327 13608 raft_consensus.cc:2258] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:42.401749 13609 ts_tablet_manager.cc:1905] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:42.402158 13607 ts_tablet_manager.cc:1905] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:42.402410 13608 ts_tablet_manager.cc:1905] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:42.407289 13017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:44:42.408053 13017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:44:42.408443 13609 ts_tablet_manager.cc:1918] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:42.408609 13609 log.cc:1196] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/bcb0096bab4f466288218cb4e283b82a
I20260925 02:44:42.409075 13609 ts_tablet_manager.cc:1939] T bcb0096bab4f466288218cb4e283b82a P e0e7ee78e54245adbbb730287ab408e6: Deleting consensus metadata
I20260925 02:44:42.409169 13608 ts_tablet_manager.cc:1918] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:42.409336 13608 log.cc:1196] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/bcb0096bab4f466288218cb4e283b82a
I20260925 02:44:42.409772 13608 ts_tablet_manager.cc:1939] T bcb0096bab4f466288218cb4e283b82a P 17c4b515cf264194ac6f229571f10055: Deleting consensus metadata
I20260925 02:44:42.410436 13004 catalog_manager.cc:4841] TS e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129:36527): tablet bcb0096bab4f466288218cb4e283b82a (table test-table [id=e89f593e0e0f46c78b3e17db6aeea608]) successfully deleted
I20260925 02:44:42.411538 13005 catalog_manager.cc:4841] TS 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313): tablet bcb0096bab4f466288218cb4e283b82a (table test-table [id=e89f593e0e0f46c78b3e17db6aeea608]) successfully deleted
I20260925 02:44:42.414633 13607 ts_tablet_manager.cc:1918] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:44:42.414826 13607 log.cc:1196] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/bcb0096bab4f466288218cb4e283b82a
I20260925 02:44:42.415306 13607 ts_tablet_manager.cc:1939] T bcb0096bab4f466288218cb4e283b82a P adbd13e8f1d6436492a3646e7d68a792: Deleting consensus metadata
I20260925 02:44:42.416544 13002 catalog_manager.cc:4841] TS adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629): tablet bcb0096bab4f466288218cb4e283b82a (table test-table [id=e89f593e0e0f46c78b3e17db6aeea608]) successfully deleted
I20260925 02:44:42.416952 13156 tablet_service.cc:1467] Processing CreateTablet for tablet ce65d8c1ddf74b36b554da95bc269c20 (DEFAULT_TABLE table=test-table [id=e20b6e76c15e4269bee7b49134c7e736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:42.417135 13296 tablet_service.cc:1467] Processing CreateTablet for tablet ce65d8c1ddf74b36b554da95bc269c20 (DEFAULT_TABLE table=test-table [id=e20b6e76c15e4269bee7b49134c7e736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:42.417402 13156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce65d8c1ddf74b36b554da95bc269c20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:42.417552 13296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce65d8c1ddf74b36b554da95bc269c20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:42.422032 13624 tablet_bootstrap.cc:492] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6: Bootstrap starting.
I20260925 02:44:42.423564 13624 tablet_bootstrap.cc:654] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:42.424924 13622 tablet_bootstrap.cc:492] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792: Bootstrap starting.
I20260925 02:44:42.425969 13438 tablet_service.cc:1467] Processing CreateTablet for tablet ce65d8c1ddf74b36b554da95bc269c20 (DEFAULT_TABLE table=test-table [id=e20b6e76c15e4269bee7b49134c7e736]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:44:42.426321 13624 tablet_bootstrap.cc:492] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6: No bootstrap required, opened a new log
I20260925 02:44:42.426483 13438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce65d8c1ddf74b36b554da95bc269c20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:42.426538 13624 ts_tablet_manager.cc:1397] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:44:42.426889 13622 tablet_bootstrap.cc:654] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:42.427534 13624 raft_consensus.cc:348] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.427786 13624 raft_consensus.cc:374] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:42.427910 13624 raft_consensus.cc:729] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0e7ee78e54245adbbb730287ab408e6, State: Initialized, Role: FOLLOWER
I20260925 02:44:42.428229 13624 consensus_queue.cc:260] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.431816 13624 ts_tablet_manager.cc:1428] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260925 02:44:42.437142 13622 tablet_bootstrap.cc:492] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792: No bootstrap required, opened a new log
I20260925 02:44:42.437372 13622 ts_tablet_manager.cc:1397] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.002s
I20260925 02:44:42.437834 13623 tablet_bootstrap.cc:492] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055: Bootstrap starting.
I20260925 02:44:42.438959 13622 raft_consensus.cc:348] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.439266 13622 raft_consensus.cc:374] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:42.439354 13622 raft_consensus.cc:729] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adbd13e8f1d6436492a3646e7d68a792, State: Initialized, Role: FOLLOWER
I20260925 02:44:42.439625 13622 consensus_queue.cc:260] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.439986 13623 tablet_bootstrap.cc:654] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:42.440194 13622 ts_tablet_manager.cc:1428] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:42.442987 13623 tablet_bootstrap.cc:492] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055: No bootstrap required, opened a new log
I20260925 02:44:42.443174 13623 ts_tablet_manager.cc:1397] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260925 02:44:42.443944 13623 raft_consensus.cc:348] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.444232 13623 raft_consensus.cc:374] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:42.444314 13623 raft_consensus.cc:729] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17c4b515cf264194ac6f229571f10055, State: Initialized, Role: FOLLOWER
I20260925 02:44:42.444550 13623 consensus_queue.cc:260] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.445109 13623 ts_tablet_manager.cc:1428] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 02:44:42.716316 13539 raft_consensus.cc:482] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:44:42.716540 13539 raft_consensus.cc:504] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.717216 13539 leader_election.cc:288] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:42.717864 13462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce65d8c1ddf74b36b554da95bc269c20" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055" is_pre_election: true
I20260925 02:44:42.717901 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce65d8c1ddf74b36b554da95bc269c20" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbd13e8f1d6436492a3646e7d68a792" is_pre_election: true
I20260925 02:44:42.718158 13462 raft_consensus.cc:2454] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 0.
I20260925 02:44:42.718209 13322 raft_consensus.cc:2454] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 0.
I20260925 02:44:42.718668 13109 leader_election.cc:302] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17c4b515cf264194ac6f229571f10055, e0e7ee78e54245adbbb730287ab408e6; no voters:
I20260925 02:44:42.719010 13539 raft_consensus.cc:2781] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:44:42.719123 13539 raft_consensus.cc:482] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:44:42.719208 13539 raft_consensus.cc:3037] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:42.721205 13539 raft_consensus.cc:504] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.721837 13539 leader_election.cc:288] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 election: Requested vote from peers adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629), 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313)
I20260925 02:44:42.722455 13462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce65d8c1ddf74b36b554da95bc269c20" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17c4b515cf264194ac6f229571f10055"
I20260925 02:44:42.722455 13322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce65d8c1ddf74b36b554da95bc269c20" candidate_uuid: "e0e7ee78e54245adbbb730287ab408e6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adbd13e8f1d6436492a3646e7d68a792"
I20260925 02:44:42.722756 13462 raft_consensus.cc:3037] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:42.722756 13322 raft_consensus.cc:3037] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:42.726346 13462 raft_consensus.cc:2454] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 1.
I20260925 02:44:42.726339 13322 raft_consensus.cc:2454] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0e7ee78e54245adbbb730287ab408e6 in term 1.
I20260925 02:44:42.727087 13106 leader_election.cc:302] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 17c4b515cf264194ac6f229571f10055, adbd13e8f1d6436492a3646e7d68a792, e0e7ee78e54245adbbb730287ab408e6; no voters:
I20260925 02:44:42.727480 13539 raft_consensus.cc:2781] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:42.727732 13539 raft_consensus.cc:686] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 1 LEADER]: Becoming Leader. State: Replica: e0e7ee78e54245adbbb730287ab408e6, State: Running, Role: LEADER
I20260925 02:44:42.728020 13539 consensus_queue.cc:237] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } }
I20260925 02:44:42.730793 13018 catalog_manager.cc:5495] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129). New cstate: current_term: 1 leader_uuid: "e0e7ee78e54245adbbb730287ab408e6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0e7ee78e54245adbbb730287ab408e6" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 } health_report { overall_health: UNKNOWN } } }
I20260925 02:44:42.827010 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:42.822163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:43483 (local address 127.5.56.129:36527)
0925 02:44:42.822438 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:42.822447 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:42.822812 (+ 365us) server_negotiation.cc:407] Connection header received
0925 02:44:42.822942 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:42.822946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:42.823009 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:42.823087 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:44:42.823425 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.824110 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.825342 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.825752 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.825842 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:42.826185 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:44:42.826340 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:44:42.826482 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:42.826681 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 02:44:42.826768 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:44:42.827190 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:42.822153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36527 (local address 127.5.56.129:43483)
0925 02:44:42.822719 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:42.822735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:42.822844 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:42.823113 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:42.823116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:42.823125 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 02:44:42.823317 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:42.823324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.824239 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.824243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:42.825215 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:42.825226 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.825881 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.825885 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:42.826006 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:42.826041 (+ 35us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 02:44:42.826500 (+ 459us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 02:44:42.826512 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 02:44:42.826650 (+ 138us) client_negotiation.cc:240] Negotiation successful
0925 02:44:42.826759 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":348,"threads_started":1}
I20260925 02:44:42.836086 13462 raft_consensus.cc:1261] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Refusing update from remote peer e0e7ee78e54245adbbb730287ab408e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:42.836154 13322 raft_consensus.cc:1261] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Refusing update from remote peer e0e7ee78e54245adbbb730287ab408e6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:44:42.836715 13578 consensus_queue.cc:1035] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "17c4b515cf264194ac6f229571f10055" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:42.837051 13598 consensus_queue.cc:1035] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adbd13e8f1d6436492a3646e7d68a792" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:44:42.889902 13018 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:42.890200 13018 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37514:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:44:42.893482 13018 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e6bcbcfc9ec14a31b5ecda5393efb8a0: Sending DeleteTablet for 3 replicas of tablet ce65d8c1ddf74b36b554da95bc269c20
I20260925 02:44:42.894476 13157 tablet_service.cc:1514] Processing DeleteTablet for tablet ce65d8c1ddf74b36b554da95bc269c20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:42 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:35256
I20260925 02:44:42.894519 13296 tablet_service.cc:1514] Processing DeleteTablet for tablet ce65d8c1ddf74b36b554da95bc269c20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:42 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:57130
I20260925 02:44:42.894833 13609 tablet_replica.cc:330] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6: stopping tablet replica
I20260925 02:44:42.894861 13438 tablet_service.cc:1514] Processing DeleteTablet for tablet ce65d8c1ddf74b36b554da95bc269c20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:44:42 UTC) from {username='oryx', principal='oryx/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:39158
I20260925 02:44:42.895042 13609 raft_consensus.cc:2229] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:44:42.895197 13607 tablet_replica.cc:330] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792: stopping tablet replica
I20260925 02:44:42.895186 13608 tablet_replica.cc:330] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055: stopping tablet replica
I20260925 02:44:42.895371 13607 raft_consensus.cc:2229] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:42.895381 13608 raft_consensus.cc:2229] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:44:42.895453 13609 raft_consensus.cc:2258] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:42.895589 13607 raft_consensus.cc:2258] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:42.895614 13608 raft_consensus.cc:2258] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:44:42.895852 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 13090
I20260925 02:44:42.896744 13609 ts_tablet_manager.cc:1905] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:42.896797 13607 ts_tablet_manager.cc:1905] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:42.897344 13608 ts_tablet_manager.cc:1905] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:44:42.904170 13609 ts_tablet_manager.cc:1918] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:44:42.904170 13607 ts_tablet_manager.cc:1918] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:44:42.904345 13607 log.cc:1196] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/ce65d8c1ddf74b36b554da95bc269c20
I20260925 02:44:42.904364 13609 log.cc:1196] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/ce65d8c1ddf74b36b554da95bc269c20
I20260925 02:44:42.904810 13609 ts_tablet_manager.cc:1939] T ce65d8c1ddf74b36b554da95bc269c20 P e0e7ee78e54245adbbb730287ab408e6: Deleting consensus metadata
I20260925 02:44:42.904810 13607 ts_tablet_manager.cc:1939] T ce65d8c1ddf74b36b554da95bc269c20 P adbd13e8f1d6436492a3646e7d68a792: Deleting consensus metadata
I20260925 02:44:42.906061 13004 catalog_manager.cc:4841] TS e0e7ee78e54245adbbb730287ab408e6 (127.5.56.129:36527): tablet ce65d8c1ddf74b36b554da95bc269c20 (table test-table [id=e20b6e76c15e4269bee7b49134c7e736]) successfully deleted
I20260925 02:44:42.906061 13002 catalog_manager.cc:4841] TS adbd13e8f1d6436492a3646e7d68a792 (127.5.56.130:45629): tablet ce65d8c1ddf74b36b554da95bc269c20 (table test-table [id=e20b6e76c15e4269bee7b49134c7e736]) successfully deleted
I20260925 02:44:42.906888 13608 ts_tablet_manager.cc:1918] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 02:44:42.907073 13608 log.cc:1196] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/ce65d8c1ddf74b36b554da95bc269c20
I20260925 02:44:42.907562 13608 ts_tablet_manager.cc:1939] T ce65d8c1ddf74b36b554da95bc269c20 P 17c4b515cf264194ac6f229571f10055: Deleting consensus metadata
I20260925 02:44:42.908672 13005 catalog_manager.cc:4841] TS 17c4b515cf264194ac6f229571f10055 (127.5.56.131:45313): tablet ce65d8c1ddf74b36b554da95bc269c20 (table test-table [id=e20b6e76c15e4269bee7b49134c7e736]) successfully deleted
Sep 25 02:44:42 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.56.129@KRBTEST.COM
I20260925 02:44:42.923638 13583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:42.897270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35292 (local address 127.5.56.129:36527)
0925 02:44:42.897462 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:42.897468 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:42.907480 (+ 10012us) server_negotiation.cc:407] Connection header received
0925 02:44:42.907618 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:42.907624 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:42.907726 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:42.907864 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:42.909200 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.910037 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.911414 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:42.911855 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:42.911960 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:42.916495 (+ 4535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:42.916516 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:42.916575 (+ 59us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:42.916660 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:42.920065 (+ 3405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:42.921144 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:42.921154 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:42.921156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:42.921334 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:42.921914 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:42.921918 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:42.921920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:42.922360 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:42.922696 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:42.923108 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 02:44:42.923304 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:44:42.924351 13222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:43.078064 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 13090
I20260925 02:44:43.106848 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 13231
Sep 25 02:44:43 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.56.130@KRBTEST.COM
I20260925 02:44:43.123358 13645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:43.107695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57150 (local address 127.5.56.130:45629)
0925 02:44:43.108198 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:43.108203 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:43.108219 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:44:43.108275 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:43.108278 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:43.108338 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:43.108431 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:43.109733 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:43.110584 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:43.111926 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:43.112385 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:43.112500 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:43.117085 (+ 4585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:43.117105 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:43.117124 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:43.117171 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:43.120348 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:43.121231 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:43.121241 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:43.121243 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:43.121387 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:43.121726 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:43.121729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:43.121731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:43.122105 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:43.122326 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:43.122717 (+ 391us) server_negotiation.cc:299] Negotiation successful
0925 02:44:43.122943 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":284,"threads_started":1}
I20260925 02:44:43.124073 13363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:43.256140 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 13231
I20260925 02:44:43.278112 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 13372
Sep 25 02:44:43 dist-test-slave-n563 krb5kdc[12973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304282, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.5.56.131@KRBTEST.COM
I20260925 02:44:43.294575 13647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:43.279059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39162 (local address 127.5.56.131:45313)
0925 02:44:43.279585 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:43.279589 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:43.279605 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:44:43.279656 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:43.279660 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:43.279717 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:43.279850 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:43.281148 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:43.281895 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:43.283268 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:43.283706 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:43.283844 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:43.288481 (+ 4637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:43.288500 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:43.288517 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:43.288568 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:43.291526 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:43.292320 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:43.292328 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:43.292330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:43.292467 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:43.292867 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:43.292873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:43.292876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:43.293263 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:43.293604 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:43.293925 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 02:44:43.294186 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":276,"threads_started":1}
I20260925 02:44:43.295354 13504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:43.394990 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 13372
I20260925 02:44:43.413789 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 12986
I20260925 02:44:43.415079 13059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:43.535626 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 12986
2026-09-25T02:44:43Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6407 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:44:43 dist-test-slave-n563 krb5kdc[13652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:44:43 dist-test-slave-n563 krb5kdc[13652](info): set up 2 sockets
Sep 25 02:44:43 dist-test-slave-n563 krb5kdc[13652](info): commencing operation
krb5kdc: starting...
W20260925 02:44:44.664923 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:44:44 dist-test-slave-n563 krb5kdc[13652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304284, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:44:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:44:44.919528 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41283
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:35561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:41283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:45.287179 13666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:45.287576 13666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:45.287747 13666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:45.295279 13666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:45.295475 13666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:45.295594 13666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:45.295704 13666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:45.295799 13666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:45.306223 13666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:35561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:41283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:45.307773 13666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:45.309279 13666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:45.322649 13671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:45.322613 13672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:45.322700 13674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:45.323676 13666 server_base.cc:1034] running on GCE node
I20260925 02:44:45.324599 13666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:45.326193 13666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:45.327445 13666 hybrid_clock.cc:648] HybridClock initialized: now 1790304285327380 us; error 60 us; skew 500 ppm
I20260925 02:44:45.327834 13666 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T02:44:45Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1864 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:44:45 dist-test-slave-n563 krb5kdc[13680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:44:45 dist-test-slave-n563 krb5kdc[13680](info): set up 2 sockets
Sep 25 02:44:45 dist-test-slave-n563 krb5kdc[13680](info): commencing operation
krb5kdc: starting...
W20260925 02:44:46.508765 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.005s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:44:46 dist-test-slave-n563 krb5kdc[13680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:44:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:44:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:44:46.750245 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:40163
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:40163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:47.099682 13693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:47.100020 13693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:47.100193 13693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:47.107915 13693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:47.108048 13693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:47.108141 13693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:47.108254 13693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:47.108338 13693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:47.118862 13693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:40163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:40163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:47.120329 13693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:47.121749 13693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:47.134332 13698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:47.134521 13699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:47.135159 13701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:47.136086 13693 server_base.cc:1034] running on GCE node
I20260925 02:44:47.136956 13693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:47.138552 13693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:47.139801 13693 hybrid_clock.cc:648] HybridClock initialized: now 1790304287139749 us; error 43 us; skew 500 ppm
I20260925 02:44:47.140188 13693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:47 dist-test-slave-n563 krb5kdc[13680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304287, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:47.143580 13693 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:44:47.145242 13693 webserver.cc:460] Webserver started at http://127.5.56.190:38163/ using document root <none> and password file <none>
I20260925 02:44:47.146034 13693 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:47.146171 13693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:47.146598 13693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:47.149860 13693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "b93315205db844ba8a31adc95417bea7"
format_stamp: "Formatted at 2026-09-25 02:44:47 on dist-test-slave-n563"
server_key: "3829c5850728989c0f4aafae38d0cd74"
server_key_iv: "8e3a652b430e6f267ba0cda02bd5a4f3"
server_key_version: "encryptionkey@0"
I20260925 02:44:47.150767 13693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "b93315205db844ba8a31adc95417bea7"
format_stamp: "Formatted at 2026-09-25 02:44:47 on dist-test-slave-n563"
server_key: "3829c5850728989c0f4aafae38d0cd74"
server_key_iv: "8e3a652b430e6f267ba0cda02bd5a4f3"
server_key_version: "encryptionkey@0"
I20260925 02:44:47.156405 13693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:44:47.160526 13708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:47.162134 13693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:47.162348 13693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "b93315205db844ba8a31adc95417bea7"
format_stamp: "Formatted at 2026-09-25 02:44:47 on dist-test-slave-n563"
server_key: "3829c5850728989c0f4aafae38d0cd74"
server_key_iv: "8e3a652b430e6f267ba0cda02bd5a4f3"
server_key_version: "encryptionkey@0"
I20260925 02:44:47.162664 13693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:47.180284 13693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:47.184693 13693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:47.185025 13693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:47.204895 13693 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:40163
I20260925 02:44:47.204929 13760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:40163 every 8 connection(s)
I20260925 02:44:47.206452 13693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:47.210654 13761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:44:47.214864 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 13693
I20260925 02:44:47.215086 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:44:47.215718 5346 external_mini_cluster.cc:1428] Setting key 1203efaf2d02b2b62560858412fae75e
I20260925 02:44:47.222496 13761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7: Bootstrap starting.
I20260925 02:44:47.226881 13761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7: Neither blocks nor log segments found. Creating new log.
I20260925 02:44:47.228603 13761 log.cc:824] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:47.233665 13761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7: No bootstrap required, opened a new log
Sep 25 02:44:47 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:47.248378 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:47.217796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.5.56.190:40163)
0925 02:44:47.218660 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:47.218671 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:47.218718 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:44:47.220785 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:47.220811 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:47.221237 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:47.221591 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:47.223536 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:47.224882 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:47.227476 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:47.229037 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:47.229250 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:47.236356 (+ 7106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:47.236376 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:47.236404 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:47.236470 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:47.241802 (+ 5332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:47.243086 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:47.243102 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:47.243115 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:47.243427 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:47.243894 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:47.243899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:47.243902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:47.244992 (+ 1090us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:47.245375 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:47.245840 (+ 465us) server_negotiation.cc:299] Negotiation successful
0925 02:44:47.246461 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":646,"thread_start_us":396,"threads_started":1}
I20260925 02:44:47.248049 13761 raft_consensus.cc:348] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } }
I20260925 02:44:47.248618 13761 raft_consensus.cc:374] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:44:47.248736 13761 raft_consensus.cc:729] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93315205db844ba8a31adc95417bea7, State: Initialized, Role: FOLLOWER
I20260925 02:44:47.249564 13761 consensus_queue.cc:260] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } }
I20260925 02:44:47.249914 13761 raft_consensus.cc:388] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:47.250085 13761 raft_consensus.cc:482] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:47.250299 13761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:44:47.253997 13761 raft_consensus.cc:504] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } }
I20260925 02:44:47.254731 13761 leader_election.cc:302] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b93315205db844ba8a31adc95417bea7; no voters:
I20260925 02:44:47.255532 13761 leader_election.cc:288] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:44:47.255705 13766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:44:47.257642 13766 raft_consensus.cc:686] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 1 LEADER]: Becoming Leader. State: Replica: b93315205db844ba8a31adc95417bea7, State: Running, Role: LEADER
I20260925 02:44:47.258276 13766 consensus_queue.cc:237] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } }
I20260925 02:44:47.258703 13761 sys_catalog.cc:564] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:47.261761 13767 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b93315205db844ba8a31adc95417bea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } } }
I20260925 02:44:47.262198 13767 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:47.262533 13768 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b93315205db844ba8a31adc95417bea7. Latest consensus state: current_term: 1 leader_uuid: "b93315205db844ba8a31adc95417bea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } } }
I20260925 02:44:47.262817 13768 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:47.271335 13773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:47.278918 13773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:47.292800 13773 catalog_manager.cc:1347] Generated new cluster ID: 07a04041addb42ab84fd63952c773d67
I20260925 02:44:47.292938 13773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:47.329523 13773 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:44:47.331048 13773 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:47.338603 13773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7: Generated new TSK 0
I20260925 02:44:47.339498 13773 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:44:47.406287 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:40163
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:47.757254 13787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:47.757627 13787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:47.757802 13787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:47.766562 13787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:47.766723 13787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:47.766898 13787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:44:47.777500 13787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:40163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:47.779067 13787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:47.780467 13787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:47.795282 13793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:47.795361 13795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:47.795385 13792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:47.796365 13787 server_base.cc:1034] running on GCE node
I20260925 02:44:47.797066 13787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:47.798115 13787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:47.799362 13787 hybrid_clock.cc:648] HybridClock initialized: now 1790304287799281 us; error 66 us; skew 500 ppm
I20260925 02:44:47.799837 13787 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:47 dist-test-slave-n563 krb5kdc[13680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304287, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:47.803334 13787 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:44:47.805043 13787 webserver.cc:460] Webserver started at http://127.5.56.129:39997/ using document root <none> and password file <none>
I20260925 02:44:47.805817 13787 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:47.805979 13787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:47.806316 13787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:47.809495 13787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "c81f1a50653943cfbbadccc318228424"
format_stamp: "Formatted at 2026-09-25 02:44:47 on dist-test-slave-n563"
server_key: "d8874de73ff0ea3a7eadfd73310663e5"
server_key_iv: "73bc9872fcfa4c6fdb3be7b2f83be9c5"
server_key_version: "encryptionkey@0"
I20260925 02:44:47.810434 13787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "c81f1a50653943cfbbadccc318228424"
format_stamp: "Formatted at 2026-09-25 02:44:47 on dist-test-slave-n563"
server_key: "d8874de73ff0ea3a7eadfd73310663e5"
server_key_iv: "73bc9872fcfa4c6fdb3be7b2f83be9c5"
server_key_version: "encryptionkey@0"
I20260925 02:44:47.816186 13787 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260925 02:44:47.820053 13802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:47.821540 13787 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:44:47.821759 13787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "c81f1a50653943cfbbadccc318228424"
format_stamp: "Formatted at 2026-09-25 02:44:47 on dist-test-slave-n563"
server_key: "d8874de73ff0ea3a7eadfd73310663e5"
server_key_iv: "73bc9872fcfa4c6fdb3be7b2f83be9c5"
server_key_version: "encryptionkey@0"
I20260925 02:44:47.822016 13787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:47.840111 13787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:47.844691 13787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:47.845013 13787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:47.846110 13787 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:47.847656 13787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:47.847736 13787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:47.847883 13787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:47.847945 13787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:47.883581 13787 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:37205
I20260925 02:44:47.883625 13916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:37205 every 8 connection(s)
I20260925 02:44:47.885044 13787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:44:47.893467 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 13787
I20260925 02:44:47.893694 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:44:47.894531 5346 external_mini_cluster.cc:1428] Setting key f2ad67cd15dac0105487d7591b2c49cf
Sep 25 02:44:47 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304287, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:47.907286 13919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:47.887791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:37421 (local address 127.5.56.190:40163)
0925 02:44:47.888276 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:47.888281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:47.889188 (+ 907us) server_negotiation.cc:407] Connection header received
0925 02:44:47.891206 (+ 2018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:47.891212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:47.891314 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:47.891439 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:47.893177 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:47.894080 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:47.896290 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:47.896706 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:47.896846 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:47.900533 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:47.900538 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:47.900541 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:47.900578 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:47.903561 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:47.904507 (+ 946us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:47.904514 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:47.904517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:47.904660 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:47.905253 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:47.905257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:47.905259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:47.905564 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:47.905688 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:47.906786 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0925 02:44:47.907024 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":256,"threads_started":1}
I20260925 02:44:47.909045 13920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:47.888252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:37421)
0925 02:44:47.889000 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:47.889046 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:47.890909 (+ 1863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:47.891713 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:47.891722 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:47.892299 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:47.892956 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:47.892985 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:47.894287 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:47.894293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:47.896134 (+ 1841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:47.896148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:47.896956 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:47.896964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:47.897211 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:47.897940 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:47.897969 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:47.900267 (+ 2298us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:47.903758 (+ 3491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:47.903770 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:47.903791 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:47.904324 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:47.904799 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:47.904808 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:47.904814 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:47.905109 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:47.905715 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:47.905728 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:47.906273 (+ 545us) client_negotiation.cc:769] Sending connection context
0925 02:44:47.906687 (+ 414us) client_negotiation.cc:240] Negotiation successful
0925 02:44:47.907388 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":305,"threads_started":1}
I20260925 02:44:47.911005 13917 heartbeater.cc:347] Connected to a master server at 127.5.56.190:40163
I20260925 02:44:47.911345 13917 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:47.912153 13917 heartbeater.cc:511] Master 127.5.56.190:40163 requested a full tablet report, sending...
I20260925 02:44:47.914605 13725 ts_manager.cc:194] Registered new tserver with Master: c81f1a50653943cfbbadccc318228424 (127.5.56.129:37205)
I20260925 02:44:47.916148 13725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:37421
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:44:47.949965 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:40163
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:48.299661 13923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:48.299978 13923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:48.300109 13923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:48.307983 13923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:48.308109 13923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:48.308322 13923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:44:48.318621 13923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:40163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:48.320024 13923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:48.321467 13923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:48.335978 13931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:48.336028 13929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:48.336064 13928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:48.336953 13923 server_base.cc:1034] running on GCE node
I20260925 02:44:48.337720 13923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:48.338889 13923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:48.340164 13923 hybrid_clock.cc:648] HybridClock initialized: now 1790304288340115 us; error 43 us; skew 500 ppm
I20260925 02:44:48.340545 13923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:48 dist-test-slave-n563 krb5kdc[13680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304288, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:48.344000 13923 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:44:48.345575 13923 webserver.cc:460] Webserver started at http://127.5.56.130:45681/ using document root <none> and password file <none>
I20260925 02:44:48.346438 13923 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:48.346598 13923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:48.346959 13923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:48.350170 13923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "cbd30473adce4965b72384d9bab97b1b"
format_stamp: "Formatted at 2026-09-25 02:44:48 on dist-test-slave-n563"
server_key: "df78de33db882c60c87d09d2da73f6dd"
server_key_iv: "500ed9a3fa13b1470a30d9b249299674"
server_key_version: "encryptionkey@0"
I20260925 02:44:48.351102 13923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "cbd30473adce4965b72384d9bab97b1b"
format_stamp: "Formatted at 2026-09-25 02:44:48 on dist-test-slave-n563"
server_key: "df78de33db882c60c87d09d2da73f6dd"
server_key_iv: "500ed9a3fa13b1470a30d9b249299674"
server_key_version: "encryptionkey@0"
I20260925 02:44:48.356675 13923 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:44:48.360620 13938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:48.362212 13923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:48.362469 13923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "cbd30473adce4965b72384d9bab97b1b"
format_stamp: "Formatted at 2026-09-25 02:44:48 on dist-test-slave-n563"
server_key: "df78de33db882c60c87d09d2da73f6dd"
server_key_iv: "500ed9a3fa13b1470a30d9b249299674"
server_key_version: "encryptionkey@0"
I20260925 02:44:48.362710 13923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:48.387216 13923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:48.392155 13923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:48.392496 13923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:48.393615 13923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:48.395066 13923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:48.395145 13923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:48.395233 13923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:48.395278 13923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:48.430692 13923 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:38813
I20260925 02:44:48.430737 14051 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:38813 every 8 connection(s)
I20260925 02:44:48.432144 13923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:44:48.434846 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 13923
I20260925 02:44:48.435056 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:44:48.435694 5346 external_mini_cluster.cc:1428] Setting key f552f419f1a2064ae25723f8f059dcf7
Sep 25 02:44:48 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304288, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
I20260925 02:44:48.459638 14054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:48.435273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:59847 (local address 127.5.56.190:40163)
0925 02:44:48.435839 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:48.435850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:48.437212 (+ 1362us) server_negotiation.cc:407] Connection header received
0925 02:44:48.440568 (+ 3356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:48.440577 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:48.440696 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:48.440859 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:48.442875 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:48.443694 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:48.446459 (+ 2765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:48.447094 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:48.447268 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:48.451626 (+ 4358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:48.451633 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:48.451637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:48.451682 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:48.455043 (+ 3361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:48.456480 (+ 1437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:48.456487 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:48.456491 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:48.456679 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:48.457319 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:48.457327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:48.457333 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:48.457701 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:48.457824 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:48.459001 (+ 1177us) server_negotiation.cc:299] Negotiation successful
0925 02:44:48.459212 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":280,"threads_started":1}
I20260925 02:44:48.460835 14055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:48.435967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:59847)
0925 02:44:48.436996 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:48.437076 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:48.440188 (+ 3112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:48.441161 (+ 973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:48.441172 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:48.441831 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:48.442635 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:48.442660 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:48.443943 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:48.443949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:48.446243 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:48.446260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:48.447397 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:48.447408 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:48.447689 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:48.448510 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:48.448539 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:48.451282 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:48.455292 (+ 4010us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:48.455308 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:48.455384 (+ 76us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:48.456247 (+ 863us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:48.456843 (+ 596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:48.456848 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:48.456853 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:48.457161 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:48.457862 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:48.457874 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:48.458463 (+ 589us) client_negotiation.cc:769] Sending connection context
0925 02:44:48.458893 (+ 430us) client_negotiation.cc:240] Negotiation successful
0925 02:44:48.459525 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"thread_start_us":389,"threads_started":1}
I20260925 02:44:48.462427 14052 heartbeater.cc:347] Connected to a master server at 127.5.56.190:40163
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
I20260925 02:44:48.462777 14052 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:48.463562 14052 heartbeater.cc:511] Master 127.5.56.190:40163 requested a full tablet report, sending...
I20260925 02:44:48.465195 13725 ts_manager.cc:194] Registered new tserver with Master: cbd30473adce4965b72384d9bab97b1b (127.5.56.130:38813)
I20260925 02:44:48.466079 13725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:59847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:44:48.489812 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:40163
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:48.843918 14058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:48.844226 14058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:48.844323 14058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:48.851684 14058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:48.851826 14058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:48.851948 14058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:44:48.862190 14058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:40163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:48.863755 14058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:48.865144 14058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:48.879405 14064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:48.879446 14066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:48.879604 14063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:48.880740 14058 server_base.cc:1034] running on GCE node
I20260925 02:44:48.881367 14058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:48.882391 14058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:48.883685 14058 hybrid_clock.cc:648] HybridClock initialized: now 1790304288883597 us; error 93 us; skew 500 ppm
I20260925 02:44:48.884060 14058 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:48 dist-test-slave-n563 krb5kdc[13680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304288, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:48.887720 14058 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:44:48.889317 14058 webserver.cc:460] Webserver started at http://127.5.56.131:36977/ using document root <none> and password file <none>
I20260925 02:44:48.890100 14058 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:48.890312 14058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:48.890688 14058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:44:48.893985 14058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "155ad1e3dd8b40d88c8a087018d69324"
format_stamp: "Formatted at 2026-09-25 02:44:48 on dist-test-slave-n563"
server_key: "4c611131129985d1d8a0e13ef2b17d35"
server_key_iv: "0f4bab134d0b45643e16091380a283b2"
server_key_version: "encryptionkey@0"
I20260925 02:44:48.894903 14058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "155ad1e3dd8b40d88c8a087018d69324"
format_stamp: "Formatted at 2026-09-25 02:44:48 on dist-test-slave-n563"
server_key: "4c611131129985d1d8a0e13ef2b17d35"
server_key_iv: "0f4bab134d0b45643e16091380a283b2"
server_key_version: "encryptionkey@0"
I20260925 02:44:48.900844 14058 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:44:48.904640 14073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:48.906075 14058 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:44:48.906268 14058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "155ad1e3dd8b40d88c8a087018d69324"
format_stamp: "Formatted at 2026-09-25 02:44:48 on dist-test-slave-n563"
server_key: "4c611131129985d1d8a0e13ef2b17d35"
server_key_iv: "0f4bab134d0b45643e16091380a283b2"
server_key_version: "encryptionkey@0"
I20260925 02:44:48.906569 14058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:48.919610 13917 heartbeater.cc:503] Master 127.5.56.190:40163 was elected leader, sending a full tablet report...
I20260925 02:44:48.924636 14058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:48.929394 14058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:48.929736 14058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:48.930831 14058 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:48.932219 14058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:48.932291 14058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:48.932461 14058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:48.932511 14058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:48.967211 14058 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:35273
I20260925 02:44:48.967247 14186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:35273 every 8 connection(s)
I20260925 02:44:48.968570 14058 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:44:48.975234 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 14058
I20260925 02:44:48.975447 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:44:48.975936 5346 external_mini_cluster.cc:1428] Setting key 664b3b1b38b3affbf28acb14d89b571f
Sep 25 02:44:48 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304288, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:44:48.989470 14189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:48.971313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:44091 (local address 127.5.56.190:40163)
0925 02:44:48.971803 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:48.971808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:48.972654 (+ 846us) server_negotiation.cc:407] Connection header received
0925 02:44:48.974560 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:48.974566 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:48.974677 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:48.974799 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:48.976608 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:48.977789 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:48.979575 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:48.979968 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:48.980076 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:48.983869 (+ 3793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:48.983876 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:48.983881 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:48.983926 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:48.986663 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:48.987369 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:48.987373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:48.987375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:48.987490 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:48.987872 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:48.987874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:48.987876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:48.988148 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:48.988266 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:48.988937 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 02:44:48.989121 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":265,"threads_started":1}
I20260925 02:44:48.990698 14190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:48.971761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:44091)
0925 02:44:48.972478 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:48.972534 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:48.974211 (+ 1677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:48.975051 (+ 840us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:48.975059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:48.975732 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:48.976375 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:48.976393 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:48.977961 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:48.977965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:48.979436 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:48.979451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:48.980162 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:48.980167 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:48.980380 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:48.981092 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:48.981117 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:48.983499 (+ 2382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:44:48.986791 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:48.986799 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:48.986813 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:48.987256 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:48.987582 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:44:48.987585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:44:48.987587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:44:48.987773 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:44:48.988252 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:44:48.988258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:44:48.988604 (+ 346us) client_negotiation.cc:769] Sending connection context
0925 02:44:48.988837 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 02:44:48.989244 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":273,"threads_started":1}
I20260925 02:44:48.992197 14187 heartbeater.cc:347] Connected to a master server at 127.5.56.190:40163
I20260925 02:44:48.992554 14187 heartbeater.cc:464] Registering TS with master...
I20260925 02:44:48.993363 14187 heartbeater.cc:511] Master 127.5.56.190:40163 requested a full tablet report, sending...
I20260925 02:44:48.994882 13725 ts_manager.cc:194] Registered new tserver with Master: 155ad1e3dd8b40d88c8a087018d69324 (127.5.56.131:35273)
I20260925 02:44:48.995738 13725 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:44091
I20260925 02:44:49.004583 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab.
Entry for principal oryx/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab.
WARNING: no policy specified for oryx/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab.
Entry for principal oryx/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab.
WARNING: no policy specified for oryx/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab.
Entry for principal oryx/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab.
I20260925 02:44:49.150166 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 13787
Sep 25 02:44:49 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:44:49.167429 14198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:49.151573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36646 (local address 127.5.56.129:37205)
0925 02:44:49.152363 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:49.152371 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:49.152407 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:44:49.152471 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:49.152477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:49.152700 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:49.152840 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:49.154187 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:49.155249 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:49.156479 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:49.157056 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:49.157161 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:49.161392 (+ 4231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:49.161406 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:49.161422 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:49.161462 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:49.164415 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:49.165156 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:49.165164 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:49.165166 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:49.165301 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:49.165697 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:49.165702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:49.165705 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:49.166100 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:49.166561 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:49.166756 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 02:44:49.167041 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":430,"threads_started":1}
I20260925 02:44:49.168777 13912 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:49.259286 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 13787
I20260925 02:44:49.278131 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 13923
Sep 25 02:44:49 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:44:49.294862 14200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:49.279118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.5.56.130:38813)
0925 02:44:49.279751 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:49.279759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:49.279788 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:44:49.279851 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:49.279858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:49.280081 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:49.280212 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:49.281462 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:49.282546 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:49.283983 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:49.284553 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:49.284655 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:49.288891 (+ 4236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:49.288904 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:49.288920 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:49.288968 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:49.292137 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:49.292842 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:49.292849 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:49.292851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:49.292976 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:49.293307 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:49.293310 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:49.293312 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:49.293662 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:49.293893 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:49.294163 (+ 270us) server_negotiation.cc:299] Negotiation successful
0925 02:44:49.294376 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":309,"threads_started":1}
I20260925 02:44:49.296172 14047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:49.381239 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 13923
I20260925 02:44:49.398711 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 14058
Sep 25 02:44:49 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:44:49.415793 14202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:49.399759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35946 (local address 127.5.56.131:35273)
0925 02:44:49.400376 (+ 617us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:49.400384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:49.400414 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:44:49.400473 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:49.400480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:49.400711 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:49.400843 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:49.402145 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:49.403292 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:49.404723 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:49.405276 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:49.405382 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:49.409987 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:49.410003 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:49.410020 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:49.410061 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:49.412939 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:49.413668 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:49.413675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:49.413677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:49.413806 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:49.414145 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:49.414148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:49.414150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:49.414688 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:49.414968 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:49.415262 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 02:44:49.415484 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":462,"thread_start_us":307,"threads_started":1}
I20260925 02:44:49.417172 14182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:49.504106 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 14058
I20260925 02:44:49.522248 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 13693
I20260925 02:44:49.523602 13756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:44:49.635686 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 13693
I20260925 02:44:49.658020 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:40163
--webserver_interface=127.5.56.190
--webserver_port=38163
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:40163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:44:50.016297 14205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:50.016727 14205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:50.016917 14205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:50.024520 14205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:44:50.024718 14205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:50.024847 14205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:50.024955 14205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:44:50.025044 14205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:44:50.035656 14205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:40163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:40163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=38163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:50.037261 14205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:50.038775 14205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:50.052608 14211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:50.052703 14210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:50.052628 14213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:50.053369 14205 server_base.cc:1034] running on GCE node
I20260925 02:44:50.054359 14205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:50.056165 14205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:50.057405 14205 hybrid_clock.cc:648] HybridClock initialized: now 1790304290057352 us; error 49 us; skew 500 ppm
I20260925 02:44:50.057832 14205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:50 dist-test-slave-n563 krb5kdc[13680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304290, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:50.061451 14205 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:44:50.063365 14205 webserver.cc:460] Webserver started at http://127.5.56.190:38163/ using document root <none> and password file <none>
I20260925 02:44:50.064180 14205 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:50.064358 14205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:50.071463 14205 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260925 02:44:50.074980 14220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:50.076598 14205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 02:44:50.076830 14205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "b93315205db844ba8a31adc95417bea7"
format_stamp: "Formatted at 2026-09-25 02:44:47 on dist-test-slave-n563"
server_key: "3829c5850728989c0f4aafae38d0cd74"
server_key_iv: "8e3a652b430e6f267ba0cda02bd5a4f3"
server_key_version: "encryptionkey@0"
I20260925 02:44:50.077955 14205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:50.101112 14205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:50.106050 14205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:50.106403 14205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:50.127729 14205 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:40163
I20260925 02:44:50.127749 14272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:40163 every 8 connection(s)
I20260925 02:44:50.129503 14205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:44:50.135104 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 14205
I20260925 02:44:50.136158 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:37205
--local_ip_for_outbound_sockets=127.5.56.129
--tserver_master_addrs=127.5.56.190:40163
--webserver_port=39997
--webserver_interface=127.5.56.129
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.56.129
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:44:50.138418 14273 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:44:50.144199 14273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7: Bootstrap starting.
I20260925 02:44:50.168156 14273 log.cc:824] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:44:50.178892 14273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:44:50.179528 14273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7: Bootstrap complete.
I20260925 02:44:50.189510 14273 raft_consensus.cc:348] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } }
I20260925 02:44:50.190604 14273 raft_consensus.cc:729] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b93315205db844ba8a31adc95417bea7, State: Initialized, Role: FOLLOWER
I20260925 02:44:50.191354 14273 consensus_queue.cc:260] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } }
I20260925 02:44:50.191648 14273 raft_consensus.cc:388] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:44:50.191787 14273 raft_consensus.cc:482] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:44:50.192003 14273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:44:50.196396 14273 raft_consensus.cc:504] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } }
I20260925 02:44:50.197089 14273 leader_election.cc:302] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b93315205db844ba8a31adc95417bea7; no voters:
I20260925 02:44:50.197831 14273 leader_election.cc:288] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:44:50.197996 14277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:44:50.199340 14277 raft_consensus.cc:686] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [term 2 LEADER]: Becoming Leader. State: Replica: b93315205db844ba8a31adc95417bea7, State: Running, Role: LEADER
I20260925 02:44:50.200062 14277 consensus_queue.cc:237] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } }
I20260925 02:44:50.200596 14273 sys_catalog.cc:564] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:44:50.202929 14279 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b93315205db844ba8a31adc95417bea7. Latest consensus state: current_term: 2 leader_uuid: "b93315205db844ba8a31adc95417bea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } } }
I20260925 02:44:50.202970 14278 sys_catalog.cc:455] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b93315205db844ba8a31adc95417bea7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b93315205db844ba8a31adc95417bea7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 40163 } } }
I20260925 02:44:50.203336 14279 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:50.203449 14278 sys_catalog.cc:458] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:44:50.208384 14283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:44:50.220314 14283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:44:50.222616 14283 catalog_manager.cc:1259] Loaded cluster ID: 07a04041addb42ab84fd63952c773d67
I20260925 02:44:50.222889 14283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:44:50.228679 14283 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:44:50.231191 14283 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b93315205db844ba8a31adc95417bea7: Loaded TSK: 0
I20260925 02:44:50.233500 14283 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:44:50.546476 14275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:50.546839 14275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:50.547037 14275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:50.554881 14275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:50.555073 14275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:50.555259 14275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:44:50.565982 14275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.129.keytab
--principal=oryx/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:37205
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=39997
--tserver_master_addrs=127.5.56.190:40163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:50.568022 14275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:50.569479 14275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:50.584139 14300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:50.584203 14302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:50.584932 14299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:50.588163 14275 server_base.cc:1034] running on GCE node
I20260925 02:44:50.588867 14275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:50.590075 14275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:50.591315 14275 hybrid_clock.cc:648] HybridClock initialized: now 1790304290591255 us; error 59 us; skew 500 ppm
I20260925 02:44:50.591650 14275 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:50 dist-test-slave-n563 krb5kdc[13680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304290, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:50.595239 14275 init.cc:377] Logged in from keytab as oryx/127.5.56.129@KRBTEST.COM (short username oryx)
I20260925 02:44:50.597138 14275 webserver.cc:460] Webserver started at http://127.5.56.129:39997/ using document root <none> and password file <none>
I20260925 02:44:50.597950 14275 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:50.598131 14275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:50.604508 14275 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:44:50.607707 14309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:50.609464 14275 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:44:50.609712 14275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "c81f1a50653943cfbbadccc318228424"
format_stamp: "Formatted at 2026-09-25 02:44:47 on dist-test-slave-n563"
server_key: "d8874de73ff0ea3a7eadfd73310663e5"
server_key_iv: "73bc9872fcfa4c6fdb3be7b2f83be9c5"
server_key_version: "encryptionkey@0"
I20260925 02:44:50.610911 14275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:50.628988 14275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:50.633744 14275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:50.634006 14275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:50.635036 14275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:50.636497 14275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:50.636587 14275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:50.636674 14275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:50.636732 14275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:50.672211 14275 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:37205
I20260925 02:44:50.672322 14421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:37205 every 8 connection(s)
I20260925 02:44:50.673861 14275 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:44:50.677438 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 14275
I20260925 02:44:50.678354 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:38813
--local_ip_for_outbound_sockets=127.5.56.130
--tserver_master_addrs=127.5.56.190:40163
--webserver_port=45681
--webserver_interface=127.5.56.130
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.56.130
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:44:50 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:50 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:44:50.694044 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:50.677185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:35337 (local address 127.5.56.190:40163)
0925 02:44:50.677954 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:50.677975 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:50.678774 (+ 799us) server_negotiation.cc:407] Connection header received
0925 02:44:50.680927 (+ 2153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:50.680942 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:50.681316 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:50.681790 (+ 474us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:50.683699 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.684986 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.686700 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.687380 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.687617 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:50.692987 (+ 5370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:35337: BlockingRecv error: recv got EOF from 127.5.56.129:35337 (error 108)
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":279,"threads_started":1}
I20260925 02:44:50.694026 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:50.677469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:35337)
0925 02:44:50.678349 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:50.678444 (+ 95us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:50.680291 (+ 1847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:50.682007 (+ 1716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:50.682023 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:50.682837 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:50.683468 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:50.683489 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.685174 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.685185 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:50.686540 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:50.686553 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.687642 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.687650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:50.687940 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:50.688695 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:50.688719 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:50.692561 (+ 3842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":353,"threads_started":1}
W20260925 02:44:50.695123 14422 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:40163 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.56.190:40163: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Sep 25 02:44:50 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:50 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:50.704360 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:50.695861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:48351)
0925 02:44:50.696052 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:50.696073 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:50.696196 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:50.696591 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:50.696596 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:50.696964 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:50.697115 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:50.697123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.698186 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.698199 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:50.699165 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:50.699178 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.699873 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.699879 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:50.700063 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:50.700754 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:50.700774 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:50.704042 (+ 3268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 02:44:50.704676 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:50.696100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:48351 (local address 127.5.56.190:40163)
0925 02:44:50.696310 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:50.696321 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:50.696335 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:44:50.696385 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:50.696389 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:50.696447 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:50.696557 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:50.697289 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.698036 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.699315 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.699725 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.699860 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:50.704281 (+ 4421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:48351: BlockingRecv error: recv got EOF from 127.5.56.129:48351 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 02:44:50 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:50 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:50.715250 14425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:50.705328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:53679)
0925 02:44:50.705529 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:50.705545 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:50.705634 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:50.705985 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:50.705988 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:50.706273 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:50.706532 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:50.706543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.708151 (+ 1608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.708157 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:50.709759 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:50.709778 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.710581 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.710587 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:50.710803 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:50.711432 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:50.711457 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:50.714889 (+ 3432us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 02:44:50.715389 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:50.705463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:53679 (local address 127.5.56.190:40163)
0925 02:44:50.705634 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:50.705650 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:50.705673 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:44:50.705744 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:50.705754 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:50.705844 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:50.705965 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:50.706691 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.707945 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.709941 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:50.710375 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:50.710541 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:50.715131 (+ 4590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:53679: BlockingRecv error: recv got EOF from 127.5.56.129:53679 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260925 02:44:50.715801 14422 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:44:51.035872 14426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:51.036190 14426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:51.036284 14426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:51.043857 14426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:51.043993 14426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:51.044185 14426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:44:51.054631 14426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.130.keytab
--principal=oryx/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:38813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=45681
--tserver_master_addrs=127.5.56.190:40163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:51.056198 14426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:51.057600 14426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:51.071811 14432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:51.071745 14434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:51.072361 14431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:51.073191 14426 server_base.cc:1034] running on GCE node
I20260925 02:44:51.073940 14426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:51.075063 14426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:51.076297 14426 hybrid_clock.cc:648] HybridClock initialized: now 1790304291076208 us; error 86 us; skew 500 ppm
I20260925 02:44:51.076658 14426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304291, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:51.080080 14426 init.cc:377] Logged in from keytab as oryx/127.5.56.130@KRBTEST.COM (short username oryx)
I20260925 02:44:51.081655 14426 webserver.cc:460] Webserver started at http://127.5.56.130:45681/ using document root <none> and password file <none>
I20260925 02:44:51.082541 14426 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:51.082693 14426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:51.088644 14426 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:44:51.091683 14441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:51.093264 14426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:44:51.093482 14426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "cbd30473adce4965b72384d9bab97b1b"
format_stamp: "Formatted at 2026-09-25 02:44:48 on dist-test-slave-n563"
server_key: "df78de33db882c60c87d09d2da73f6dd"
server_key_iv: "500ed9a3fa13b1470a30d9b249299674"
server_key_version: "encryptionkey@0"
I20260925 02:44:51.094617 14426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:51.107957 14426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:51.113032 14426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:51.113370 14426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:51.114552 14426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:51.116086 14426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:51.116175 14426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:51.116259 14426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:51.116313 14426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:51.152469 14426 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:38813
I20260925 02:44:51.152534 14556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:38813 every 8 connection(s)
I20260925 02:44:51.154032 14426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:44:51.163710 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 14426
I20260925 02:44:51.164346 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:35273
--local_ip_for_outbound_sockets=127.5.56.131
--tserver_master_addrs=127.5.56.190:40163
--webserver_port=36977
--webserver_interface=127.5.56.131
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:34009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.5.56.131
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:44:51.171518 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:51.156913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:50229 (local address 127.5.56.190:40163)
0925 02:44:51.157116 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:51.157121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:51.158422 (+ 1301us) server_negotiation.cc:407] Connection header received
0925 02:44:51.160433 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:51.160439 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:51.160541 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:51.160677 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:51.162447 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.163177 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.165057 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.165548 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.165701 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.171246 (+ 5545us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:50229: BlockingRecv error: recv got EOF from 127.5.56.130:50229 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:44:51.172338 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:51.157419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:50229)
0925 02:44:51.158217 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:51.158259 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:51.160107 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:51.160954 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:51.160964 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:51.161575 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:51.162207 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.162228 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.163383 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.163387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.164871 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.164886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.165828 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.165836 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.166096 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.166890 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:51.166915 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:51.170998 (+ 4083us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":309,"threads_started":1}
W20260925 02:44:51.173106 14557 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:40163 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.56.190:40163: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:51.182430 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:51.173839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:37191)
0925 02:44:51.174062 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:51.174084 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:51.174222 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:51.174620 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:51.174625 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:51.175002 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:51.175157 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.175166 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.176407 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.176411 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.177332 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.177345 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.178107 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.178113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.178294 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.178993 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:51.179014 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:51.182090 (+ 3076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 02:44:51.182657 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:51.173938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:37191 (local address 127.5.56.190:40163)
0925 02:44:51.174145 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:51.174151 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:51.174173 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:44:51.174357 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:51.174363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:51.174468 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:51.174589 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:51.175318 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.176248 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.177486 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.177916 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.178087 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.182361 (+ 4274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:37191: BlockingRecv error: recv got EOF from 127.5.56.130:37191 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:51.191776 14559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:51.183485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:52021)
0925 02:44:51.183688 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:51.183708 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:51.183838 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:51.184199 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:51.184204 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:51.184548 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:51.184709 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.184718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.185751 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.185757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.186787 (+ 1030us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.186802 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.187576 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.187582 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.187759 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.188413 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:51.188432 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:51.191424 (+ 2992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260925 02:44:51.191927 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:51.183567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:52021 (local address 127.5.56.190:40163)
0925 02:44:51.183738 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:51.183746 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:51.183767 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:44:51.183970 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:51.183975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:51.184060 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:51.184164 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:51.184860 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.185604 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.186939 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.187406 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.187550 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.191686 (+ 4136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:52021: BlockingRecv error: recv got EOF from 127.5.56.130:52021 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260925 02:44:51.192297 14557 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:44:51.535000 14560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:44:51.535423 14560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:44:51.535610 14560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:44:51.543736 14560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:44:51.543962 14560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:44:51.544183 14560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:44:51.555009 14560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:34009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/krb5kdc/oryx_127.5.56.131.keytab
--principal=oryx/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:35273
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=36977
--tserver_master_addrs=127.5.56.190:40163
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:44:51.556670 14560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:44:51.558240 14560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:44:51.573930 14566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:51.573977 14565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:44:51.574012 14568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:44:51.575528 14560 server_base.cc:1034] running on GCE node
I20260925 02:44:51.576310 14560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:44:51.577512 14560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:44:51.578785 14560 hybrid_clock.cc:648] HybridClock initialized: now 1790304291578704 us; error 78 us; skew 500 ppm
I20260925 02:44:51.579206 14560 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304291, etypes {rep=17 tkt=17 ses=17}, oryx/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:44:51.582922 14560 init.cc:377] Logged in from keytab as oryx/127.5.56.131@KRBTEST.COM (short username oryx)
I20260925 02:44:51.584707 14560 webserver.cc:460] Webserver started at http://127.5.56.131:36977/ using document root <none> and password file <none>
I20260925 02:44:51.585489 14560 fs_manager.cc:359] Metadata directory not provided
I20260925 02:44:51.585620 14560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:44:51.592336 14560 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260925 02:44:51.595574 14575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:51.597370 14560 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:44:51.597581 14560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "155ad1e3dd8b40d88c8a087018d69324"
format_stamp: "Formatted at 2026-09-25 02:44:48 on dist-test-slave-n563"
server_key: "4c611131129985d1d8a0e13ef2b17d35"
server_key_iv: "0f4bab134d0b45643e16091380a283b2"
server_key_version: "encryptionkey@0"
I20260925 02:44:51.598979 14560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:44:51.627465 14560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:44:51.632485 14560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:44:51.632745 14560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:44:51.633790 14560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:44:51.635324 14560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:44:51.635433 14560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:51.635522 14560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:44:51.635581 14560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:44:51.671449 14560 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:35273
I20260925 02:44:51.671686 14689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:35273 every 8 connection(s)
I20260925 02:44:51.673130 14560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:44:51.681360 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 14560
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
W20260925 02:44:51.692252 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:51.676099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:46351 (local address 127.5.56.190:40163)
0925 02:44:51.676326 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:51.676334 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:51.677829 (+ 1495us) server_negotiation.cc:407] Connection header received
0925 02:44:51.679877 (+ 2048us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:51.679882 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:51.679993 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:51.680090 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:51.682022 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.682983 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.685374 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.685784 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.685981 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.691893 (+ 5912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:46351: BlockingRecv error: recv got EOF from 127.5.56.131:46351 (error 108)
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":79}
I20260925 02:44:51.692938 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:51.676671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:46351)
0925 02:44:51.677629 (+ 958us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:51.677677 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:51.679552 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:51.680325 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:51.680333 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:51.680977 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:51.681795 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.681816 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.683185 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.683191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.685199 (+ 2008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.685221 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.686050 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.686056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.686276 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.687094 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:51.687119 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:51.691562 (+ 4443us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":363,"threads_started":1}
W20260925 02:44:51.693826 14690 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:40163 (0 consecutive failures): Not authorized: Failed to ping master at 127.5.56.190:40163: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
I20260925 02:44:51.701095 14694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:51.683231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39968 (local address 127.5.56.190:40163)
0925 02:44:51.683721 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:51.683725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:51.683739 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:44:51.683780 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:51.683783 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:51.683839 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:51.683925 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:51.685925 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.687119 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.688980 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.689470 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.689584 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.692961 (+ 3377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:44:51.692975 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:44:51.693000 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:44:51.693058 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:44:51.697279 (+ 4221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:51.698151 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:51.698159 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:51.698167 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:51.698436 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:44:51.698879 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:44:51.698882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:44:51.698885 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:44:51.699530 (+ 645us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:44:51.699841 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 02:44:51.700308 (+ 467us) server_negotiation.cc:299] Negotiation successful
0925 02:44:51.700785 (+ 477us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":405,"thread_start_us":291,"threads_started":1}
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:51.705363 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:51.694647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:40869)
0925 02:44:51.694835 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:51.694854 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:51.694997 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:51.695427 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:51.695431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:51.695755 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:51.695914 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.695923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.698079 (+ 2156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.698087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.699401 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.699415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.700396 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.700403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.700644 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.701308 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:51.701334 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:51.705043 (+ 3709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 02:44:51.705521 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:51.694706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:40869 (local address 127.5.56.190:40163)
0925 02:44:51.694914 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:51.694921 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:51.694945 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:44:51.695148 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:51.695151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:51.695251 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:51.695369 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:51.696078 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.697278 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.699539 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.700189 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.700313 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.705263 (+ 4950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:40869: BlockingRecv error: recv got EOF from 127.5.56.131:40869 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:51.714102 14692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:51.706220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:34413)
0925 02:44:51.706428 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:51.706447 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:51.706546 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:51.706804 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:51.706807 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:51.707134 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:51.707295 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.707303 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.708293 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.708297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.709349 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.709361 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.710025 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.710029 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.710151 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.710812 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:51.710829 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:51.713769 (+ 2940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 02:44:51.714329 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:51.706280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:34413 (local address 127.5.56.190:40163)
0925 02:44:51.706539 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:51.706543 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:51.706559 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:44:51.706628 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:51.706631 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:51.706689 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:51.706767 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:51.707423 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.708171 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.709473 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.709897 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.709991 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.714028 (+ 4037us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:34413: BlockingRecv error: recv got EOF from 127.5.56.131:34413 (error 108)
Metrics: {"server-negotiator.queue_time_us":145}
W20260925 02:44:51.714709 14690 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:51 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:51.725849 14695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:51.716393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:49991)
0925 02:44:51.717250 (+ 857us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:51.717267 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:51.717436 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:51.717786 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:51.717791 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:51.718460 (+ 669us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:51.718691 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.718701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.719766 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.719772 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.720753 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:51.720766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.721553 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.721560 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:51.721754 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.722470 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:51.722494 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:51.725495 (+ 3001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":565,"threads_started":1}
W20260925 02:44:51.725998 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:51.716477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:49991 (local address 127.5.56.190:40163)
0925 02:44:51.716675 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:51.716682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:51.717353 (+ 671us) server_negotiation.cc:407] Connection header received
0925 02:44:51.717576 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:51.717581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:51.717644 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:51.717749 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:51.718835 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.719603 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.720906 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:51.721354 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:51.721509 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:51.725761 (+ 4252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:49991: BlockingRecv error: recv got EOF from 127.5.56.129:49991 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 25 02:44:52 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:52 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:52.202771 14696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:52.192954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:35175)
0925 02:44:52.193463 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:52.193481 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:52.193608 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:52.193927 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:52.193931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:52.194507 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:52.194719 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:52.194729 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.195764 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.195768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:52.196753 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:52.196766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.197611 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.197616 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:52.197751 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:52.198576 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:52.198620 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:52.202348 (+ 3728us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":280,"threads_started":1}
W20260925 02:44:52.203022 14424 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:52.193063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:35175 (local address 127.5.56.190:40163)
0925 02:44:52.193353 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:52.193360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:52.193573 (+ 213us) server_negotiation.cc:407] Connection header received
0925 02:44:52.193722 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:52.193726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:52.193793 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:52.193877 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:52.194873 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.195622 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.196904 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.197370 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.197508 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:52.202683 (+ 5175us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:35175: BlockingRecv error: recv got EOF from 127.5.56.130:35175 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 25 02:44:52 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:52 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:52.728477 14697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:52.715456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:50505)
0925 02:44:52.716141 (+ 685us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:52.716159 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:52.716293 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:52.716633 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:52.716636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:52.717149 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:52.717348 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:52.717357 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.718476 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.718480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:52.719435 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:52.719447 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.720226 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.720233 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:52.720439 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:52.724687 (+ 4248us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:52.724710 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:52.728137 (+ 3427us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":381,"threads_started":1}
W20260925 02:44:52.728652 14698 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:52.715621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:50505 (local address 127.5.56.190:40163)
0925 02:44:52.716163 (+ 542us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:52.716168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:52.716243 (+ 75us) server_negotiation.cc:407] Connection header received
0925 02:44:52.716403 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:52.716407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:52.716488 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:52.716585 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:52.717517 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.718279 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.719590 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.720018 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.720119 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:52.728391 (+ 8272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:50505: BlockingRecv error: recv got EOF from 127.5.56.131:50505 (error 108)
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":292,"threads_started":1}
Sep 25 02:44:52 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:52 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:52.736493 14699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:52.727098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:48493)
0925 02:44:52.727727 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:52.727748 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:52.727878 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:52.728760 (+ 882us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:52.728766 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:52.729193 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:52.729345 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:52.729354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.730487 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.730492 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:52.731406 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:52.731418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.732169 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.732175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:52.732392 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:52.733056 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:52.733082 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:52.736221 (+ 3139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":342,"threads_started":1}
W20260925 02:44:52.736766 14700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:52.727123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:48493 (local address 127.5.56.190:40163)
0925 02:44:52.728394 (+ 1271us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:52.728400 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:52.728418 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:44:52.728480 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:52.728484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:52.728565 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:52.728689 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:52.729508 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.730319 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.731574 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:52.732000 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:52.732151 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:52.736453 (+ 4302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:48493: BlockingRecv error: recv got EOF from 127.5.56.129:48493 (error 108)
Metrics: {"server-negotiator.queue_time_us":1159,"thread_start_us":989,"threads_started":1}
Sep 25 02:44:53 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:53 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:53.212913 14701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:53.203757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:58709)
0925 02:44:53.204357 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:53.204374 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:53.204556 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:53.204936 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:53.204942 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:53.205398 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:53.205553 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:53.205563 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.206702 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.206707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:53.207639 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:53.207652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.208384 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.208390 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:53.208562 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:53.209304 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:53.209324 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:53.212539 (+ 3215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":346,"threads_started":1}
W20260925 02:44:53.213014 14700 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:53.203899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:58709 (local address 127.5.56.190:40163)
0925 02:44:53.204134 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:53.204143 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:53.204489 (+ 346us) server_negotiation.cc:407] Connection header received
0925 02:44:53.204706 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:53.204715 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:53.204784 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:53.204895 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:53.205716 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.206541 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.207802 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.208217 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.208356 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:53.212787 (+ 4431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:58709: BlockingRecv error: recv got EOF from 127.5.56.130:58709 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 02:44:53 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:53 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:53.738279 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:53.729619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:55307)
0925 02:44:53.730133 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:53.730150 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:53.730256 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:53.730811 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:53.730815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:53.731232 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:53.731378 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:53.731386 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.732360 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.732364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:53.733267 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:53.733278 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.733926 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.733930 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:53.734046 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:53.734641 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:53.734659 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:53.737793 (+ 3134us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":274,"threads_started":1}
W20260925 02:44:53.739245 14703 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:53.729818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:55307 (local address 127.5.56.190:40163)
0925 02:44:53.730503 (+ 685us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:53.730507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:53.730523 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:44:53.730571 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:53.730574 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:53.730641 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:53.730745 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:53.731509 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.732235 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.733392 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.733798 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.733886 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:53.738850 (+ 4964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:55307: BlockingRecv error: recv got EOF from 127.5.56.131:55307 (error 108)
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":406,"threads_started":1}
Sep 25 02:44:53 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:53 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:53.746745 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:53.737522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:54747)
0925 02:44:53.738669 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:53.738689 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:53.738831 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:53.739193 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:53.739198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:53.739670 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:53.739863 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:53.739872 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.741088 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.741094 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:53.741998 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:53.742009 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.742744 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.742750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:53.742948 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:53.743530 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:53.743548 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:53.746365 (+ 2817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1031,"thread_start_us":897,"threads_started":1}
W20260925 02:44:53.746954 14705 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:53.737660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:54747 (local address 127.5.56.190:40163)
0925 02:44:53.738487 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:53.738495 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:53.738854 (+ 359us) server_negotiation.cc:407] Connection header received
0925 02:44:53.738942 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:53.738946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:53.739033 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:53.739161 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:53.740015 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.740957 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.742136 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:53.742590 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:53.742721 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:53.746649 (+ 3928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:54747: BlockingRecv error: recv got EOF from 127.5.56.129:54747 (error 108)
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":498,"threads_started":1}
Sep 25 02:44:54 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:54 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:54.229048 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:54.214216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:57399)
0925 02:44:54.214844 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:54.214871 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:54.214990 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:54.215322 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:54.215325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:54.215699 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:54.215869 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:54.215878 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.216879 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.216883 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:54.217806 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:54.217818 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.218546 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.218551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:54.218673 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:54.225800 (+ 7127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:54.225838 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:54.228773 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":360,"threads_started":1}
W20260925 02:44:54.229254 14705 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:54.214288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:57399 (local address 127.5.56.190:40163)
0925 02:44:54.214518 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:54.214524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:54.214955 (+ 431us) server_negotiation.cc:407] Connection header received
0925 02:44:54.215102 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:54.215105 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:54.215171 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:54.215258 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:54.216009 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.216749 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.217941 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.218333 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.218502 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:54.229030 (+ 10528us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:57399: BlockingRecv error: recv got EOF from 127.5.56.130:57399 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 02:44:54 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:54 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:54.749399 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:54.739677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:57875)
0925 02:44:54.740202 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:54.740219 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:54.740336 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:54.740851 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:54.740855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:54.741244 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:54.741475 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:54.741485 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.742900 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.742905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:54.744146 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:54.744159 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.744822 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.744826 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:54.744957 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:54.745607 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:54.745626 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:54.749048 (+ 3422us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":251,"threads_started":1}
W20260925 02:44:54.749610 14708 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:54.739795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:57875 (local address 127.5.56.190:40163)
0925 02:44:54.740541 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:54.740546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:54.740564 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:44:54.740617 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:54.740621 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:54.740693 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:54.740785 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:54.741633 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.742708 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.744288 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.744694 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.744791 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:54.749281 (+ 4490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:57875: BlockingRecv error: recv got EOF from 127.5.56.131:57875 (error 108)
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":514,"threads_started":1}
Sep 25 02:44:54 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:54 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:54.756780 14709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:54.747740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:43303)
0925 02:44:54.748308 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:54.748344 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:54.748493 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:54.748922 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:54.748930 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:54.749339 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:54.749548 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:54.749565 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.750814 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.750818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:54.751772 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:54.751783 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.752469 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.752473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:54.752608 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:54.753233 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:54.753251 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:54.756546 (+ 3295us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":288,"threads_started":1}
W20260925 02:44:54.757071 14710 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:54.747815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:43303 (local address 127.5.56.190:40163)
0925 02:44:54.748637 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:54.748642 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:54.748656 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:44:54.748701 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:54.748704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:54.748759 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:54.748885 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:54.749730 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.750681 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.751919 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:54.752342 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:54.752454 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:54.756760 (+ 4306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:43303: BlockingRecv error: recv got EOF from 127.5.56.129:43303 (error 108)
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":522,"threads_started":1}
Sep 25 02:44:55 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:55 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:55.240659 14711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:55.230248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:48063)
0925 02:44:55.230847 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:55.230870 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:55.231081 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:55.231605 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:55.231608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:55.232058 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:55.232260 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:55.232270 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.233705 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.233711 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:55.235536 (+ 1825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:55.235560 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.236520 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.236526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:55.236725 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:55.237347 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:55.237366 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:55.240384 (+ 3018us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":437,"threads_started":1}
W20260925 02:44:55.240890 14710 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:55.230623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:48063 (local address 127.5.56.190:40163)
0925 02:44:55.230813 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:55.230818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:55.230993 (+ 175us) server_negotiation.cc:407] Connection header received
0925 02:44:55.231341 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:55.231347 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:55.231440 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:55.231565 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:55.232471 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.233548 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.235748 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.236350 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.236505 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:55.240620 (+ 4115us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:48063: BlockingRecv error: recv got EOF from 127.5.56.130:48063 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 25 02:44:55 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:55 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:55.759888 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:55.750520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:37777)
0925 02:44:55.751036 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:55.751053 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:55.751168 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:55.751644 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:55.751647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:55.752043 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:55.752197 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:55.752205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.753173 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.753177 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:55.754143 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:55.754155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.755047 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.755053 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:55.755199 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:55.755817 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:55.755835 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:55.759530 (+ 3695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":278,"threads_started":1}
W20260925 02:44:55.760761 14713 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:55.750791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:37777 (local address 127.5.56.190:40163)
0925 02:44:55.751329 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:55.751333 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:55.751349 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:44:55.751398 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:55.751401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:55.751461 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:55.751567 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:55.752334 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.753058 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.754273 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.754825 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.754929 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:55.760445 (+ 5516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:37777: BlockingRecv error: recv got EOF from 127.5.56.131:37777 (error 108)
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":298,"threads_started":1}
Sep 25 02:44:55 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:55 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:55.768137 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:55.758097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:59773)
0925 02:44:55.758743 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:55.758764 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:55.758899 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:55.759867 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:55.759873 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:55.760502 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:55.760751 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:55.760764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.762184 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.762189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:55.763253 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:55.763268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.764044 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.764048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:55.764176 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:55.764761 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:55.764780 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:55.767864 (+ 3084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":345,"threads_started":1}
W20260925 02:44:55.768491 14715 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:55.758193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:59773 (local address 127.5.56.190:40163)
0925 02:44:55.759546 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:55.759552 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:55.759571 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:44:55.759628 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:55.759632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:55.759703 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:55.759818 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:55.761186 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.762010 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.763407 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:55.763878 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:55.763984 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:55.768153 (+ 4169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:59773: BlockingRecv error: recv got EOF from 127.5.56.129:59773 (error 108)
Metrics: {"mutex_wait_us":414,"server-negotiator.queue_time_us":1183,"thread_start_us":1011,"threads_started":1}
Sep 25 02:44:56 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:56 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:56.250782 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:56.241875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:38671)
0925 02:44:56.242567 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:56.242584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:56.242746 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:56.243100 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:56.243105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:56.243544 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:56.243704 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:56.243713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.244814 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.244821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:56.245747 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:56.245759 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.246501 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.246507 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:56.246690 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:56.247293 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:56.247311 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:56.250374 (+ 3063us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":392,"threads_started":1}
W20260925 02:44:56.250898 14715 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:56.242160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:38671 (local address 127.5.56.190:40163)
0925 02:44:56.242436 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:56.242444 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:56.242674 (+ 230us) server_negotiation.cc:407] Connection header received
0925 02:44:56.242881 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:56.242886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:56.242957 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:56.243067 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:56.243861 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.244667 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.245895 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.246308 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.246479 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:56.250656 (+ 4177us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:38671: BlockingRecv error: recv got EOF from 127.5.56.130:38671 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 25 02:44:56 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:56 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:56.770112 14717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:56.760944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:40883)
0925 02:44:56.761565 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:56.761583 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:56.761696 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:56.762041 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:56.762044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:56.762476 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:56.762659 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:56.762669 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.763712 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.763716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:56.764631 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:56.764642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.765590 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.765597 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:56.765779 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:56.766639 (+ 860us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:56.766657 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:56.769723 (+ 3066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":316,"threads_started":1}
W20260925 02:44:56.770579 14718 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:56.761119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:40883 (local address 127.5.56.190:40163)
0925 02:44:56.761639 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:56.761644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:56.761663 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:44:56.761814 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:56.761820 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:56.761889 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:56.761991 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:56.762821 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.763580 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.764801 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.765361 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.765506 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:56.769971 (+ 4465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:40883: BlockingRecv error: recv got EOF from 127.5.56.131:40883 (error 108)
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":304,"threads_started":1}
Sep 25 02:44:56 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:56 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:56.779001 14719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:56.769107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:56529)
0925 02:44:56.770923 (+ 1816us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:56.770943 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:56.771091 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:56.771473 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:56.771479 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:56.771879 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:56.772034 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:56.772042 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.773098 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.773103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:56.774013 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:56.774024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.774837 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.774842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:56.775027 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:56.775619 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:56.775637 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:56.778687 (+ 3050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1683,"thread_start_us":1553,"threads_started":1}
W20260925 02:44:56.779320 14718 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:56.769172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:56529 (local address 127.5.56.190:40163)
0925 02:44:56.770990 (+ 1818us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:56.770997 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:56.771016 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:44:56.771233 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:56.771237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:56.771300 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:56.771438 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:56.772180 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.772926 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.774198 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:56.774669 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:56.774810 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:56.778861 (+ 4051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:56529: BlockingRecv error: recv got EOF from 127.5.56.129:56529 (error 108)
Metrics: {"server-negotiator.queue_time_us":1709,"thread_start_us":1863,"threads_started":1}
Sep 25 02:44:57 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:57 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:57.261255 14721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:57.251896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:49461)
0925 02:44:57.252507 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:57.252527 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:57.252674 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:57.253014 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:57.253017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:57.253414 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:57.253569 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:57.253577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.254778 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.254783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:57.255707 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:57.255721 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.256503 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.256509 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:57.256641 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:57.257301 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:57.257320 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:57.260878 (+ 3558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":368,"threads_started":1}
W20260925 02:44:57.261423 14718 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:57.252051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:49461 (local address 127.5.56.190:40163)
0925 02:44:57.252244 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:57.252249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:57.252702 (+ 453us) server_negotiation.cc:407] Connection header received
0925 02:44:57.252778 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:57.252782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:57.252863 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:57.252970 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:57.253710 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.254611 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.255871 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.256331 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.256434 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:57.261150 (+ 4716us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:49461: BlockingRecv error: recv got EOF from 127.5.56.130:49461 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 02:44:57 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:57 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:57.786710 14722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:57.771129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:34979)
0925 02:44:57.771746 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:57.771762 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:57.771885 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:57.772214 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:57.772218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:57.772638 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:57.772799 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:57.772809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.773864 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.773868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:57.774804 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:57.774817 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.775569 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.775573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:57.775698 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:57.782681 (+ 6983us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:57.782703 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:57.786362 (+ 3659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":327,"threads_started":1}
W20260925 02:44:57.787025 14723 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:57.771206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:34979 (local address 127.5.56.190:40163)
0925 02:44:57.771732 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:57.771736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:57.771839 (+ 103us) server_negotiation.cc:407] Connection header received
0925 02:44:57.771987 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:57.771991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:57.772068 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:57.772162 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:57.773013 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.773732 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.774967 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.775403 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.775513 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:57.786696 (+ 11183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:34979: BlockingRecv error: recv got EOF from 127.5.56.131:34979 (error 108)
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":295,"threads_started":1}
Sep 25 02:44:57 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:57 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:57.790587 14724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:57.779984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:35479)
0925 02:44:57.781137 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:57.781162 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:57.781296 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:57.781673 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:57.781678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:57.782161 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:57.782481 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:57.782497 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.783836 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.783844 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:57.785352 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:57.785367 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.786373 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.786380 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:57.786610 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:57.787365 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:57.787385 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:57.790250 (+ 2865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":605,"threads_started":1}
W20260925 02:44:57.791051 14725 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:57.780052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:35479 (local address 127.5.56.190:40163)
0925 02:44:57.781276 (+ 1224us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:57.781282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:57.781301 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:44:57.781430 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:57.781435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:57.781505 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:57.781617 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:57.782646 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.783655 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.785504 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:57.786166 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:57.786308 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:57.790604 (+ 4296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:35479: BlockingRecv error: recv got EOF from 127.5.56.129:35479 (error 108)
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":948,"threads_started":1}
Sep 25 02:44:58 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:58 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:58.271097 14726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:58.262512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:53405)
0925 02:44:58.263129 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:58.263146 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:58.263267 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:58.263646 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:58.263649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:58.264111 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:58.264281 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:58.264290 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.265328 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.265333 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:58.266290 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:58.266302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.267059 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.267064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:58.267189 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:58.267780 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:58.267798 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:58.270831 (+ 3033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":296,"threads_started":1}
W20260925 02:44:58.271377 14725 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:58.262556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:53405 (local address 127.5.56.190:40163)
0925 02:44:58.262783 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:58.262787 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:58.263225 (+ 438us) server_negotiation.cc:407] Connection header received
0925 02:44:58.263396 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:58.263402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:58.263490 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:58.263599 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:58.264417 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.265182 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.266495 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.266911 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.267030 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:58.271140 (+ 4110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:53405: BlockingRecv error: recv got EOF from 127.5.56.130:53405 (error 108)
Metrics: {"server-negotiator.queue_time_us":113}
Sep 25 02:44:58 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:58 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:58.798169 14727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:58.787728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:59589)
0925 02:44:58.788597 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:58.788620 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:58.788777 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:58.789216 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:58.789220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:58.789702 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:58.789871 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:58.789881 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.791029 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.791034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:58.792552 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:58.792573 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.793632 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.793639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:58.793782 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:58.794454 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:58.794481 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:58.797781 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":724,"thread_start_us":531,"threads_started":1}
W20260925 02:44:58.798452 14728 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:58.787814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:59589 (local address 127.5.56.190:40163)
0925 02:44:58.788865 (+ 1051us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:58.788873 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:58.788891 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:44:58.788951 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:58.788954 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:58.789028 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:58.789158 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:58.790017 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.790862 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.792747 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.793474 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.793607 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:58.798049 (+ 4442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:59589: BlockingRecv error: recv got EOF from 127.5.56.131:59589 (error 108)
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":816,"threads_started":1}
Sep 25 02:44:58 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:58 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:58.802753 14729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:58.791590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:33373)
0925 02:44:58.793342 (+ 1752us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:58.793358 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:58.793550 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:58.793938 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:58.793943 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:58.794527 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:58.794731 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:58.794743 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.796008 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.796013 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:58.797286 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:58.797300 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.798116 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.798123 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:58.798336 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:58.799110 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:58.799129 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:58.802453 (+ 3324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1620,"thread_start_us":1516,"threads_started":1}
W20260925 02:44:58.802896 14730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:58.791754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:33373 (local address 127.5.56.190:40163)
0925 02:44:58.793195 (+ 1441us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:58.793201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:58.793455 (+ 254us) server_negotiation.cc:407] Connection header received
0925 02:44:58.793692 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:58.793696 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:58.793766 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:58.793856 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:58.794888 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.795841 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.797442 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:58.797917 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:58.798078 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:58.802660 (+ 4582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:33373: BlockingRecv error: recv got EOF from 127.5.56.129:33373 (error 108)
Metrics: {"server-negotiator.queue_time_us":1306,"thread_start_us":1163,"threads_started":1}
Sep 25 02:44:59 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:59 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:59.281108 14731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:59.272027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:50053)
0925 02:44:59.272658 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:59.272674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:59.272900 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:59.273291 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:59.273295 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:59.273743 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:59.273902 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:59.273911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.275212 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.275219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:59.276125 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:59.276136 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.276892 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.276898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:59.277065 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:59.277736 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:59.277755 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:59.280768 (+ 3013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":339,"threads_started":1}
W20260925 02:44:59.281260 14730 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:59.272199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:50053 (local address 127.5.56.190:40163)
0925 02:44:59.272478 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:59.272485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:59.272766 (+ 281us) server_negotiation.cc:407] Connection header received
0925 02:44:59.273064 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:59.273070 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:59.273149 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:59.273257 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:59.274050 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.275068 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.276281 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.276698 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.276843 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:59.280996 (+ 4153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:50053: BlockingRecv error: recv got EOF from 127.5.56.130:50053 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 25 02:44:59 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:59 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:59.810348 14732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:59.799402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:46097)
0925 02:44:59.800086 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:59.800106 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:59.800233 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:59.800662 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:59.800665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:59.801098 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:59.801255 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:59.801264 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.802590 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.802598 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:59.804059 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:59.804076 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.805345 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.805349 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:59.805473 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:59.806179 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:59.806206 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:59.809916 (+ 3710us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":410,"threads_started":1}
W20260925 02:44:59.810379 14733 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:59.799557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:46097 (local address 127.5.56.190:40163)
0925 02:44:59.800057 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:59.800063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:59.800193 (+ 130us) server_negotiation.cc:407] Connection header received
0925 02:44:59.800341 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:59.800345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:59.800432 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:59.800537 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:59.801417 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.802364 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.804238 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.804953 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.805051 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:59.810087 (+ 5036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:46097: BlockingRecv error: recv got EOF from 127.5.56.131:46097 (error 108)
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":261,"threads_started":1}
Sep 25 02:44:59 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:44:59 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:44:59.814004 14734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:44:59.803809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:58989)
0925 02:44:59.804481 (+ 672us) negotiation.cc:107] Waiting for socket to connect
0925 02:44:59.804501 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:44:59.804622 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:44:59.805062 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:44:59.805065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:44:59.805454 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:44:59.805646 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:59.805655 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.807168 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.807173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:59.808718 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:44:59.808739 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.809716 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.809723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:44:59.809981 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:59.810783 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:44:59.810807 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:44:59.813626 (+ 2819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":368,"threads_started":1}
W20260925 02:44:59.814148 14735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:44:59.804168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:58989 (local address 127.5.56.190:40163)
0925 02:44:59.804745 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0925 02:44:59.804752 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:44:59.804771 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:44:59.804822 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:44:59.804827 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:44:59.804906 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:44:59.805021 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:44:59.805817 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.807018 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.808888 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:44:59.809546 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:44:59.809641 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:44:59.813855 (+ 4214us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:58989: BlockingRecv error: recv got EOF from 127.5.56.129:58989 (error 108)
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":272,"threads_started":1}
Sep 25 02:45:00 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:00 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:00.291400 14736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:00.282132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:55421)
0925 02:45:00.282731 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:00.282747 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:00.282861 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:00.283246 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:00.283252 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:00.283700 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:00.283861 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:00.283871 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.284928 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.284936 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:00.285909 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:00.285921 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.286727 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.286733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:00.286911 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:00.287553 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:00.287571 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:00.290989 (+ 3418us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":365,"threads_started":1}
W20260925 02:45:00.291487 14735 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:00.282206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:55421 (local address 127.5.56.190:40163)
0925 02:45:00.282442 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:00.282454 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:00.282824 (+ 370us) server_negotiation.cc:407] Connection header received
0925 02:45:00.283008 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:00.283019 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:00.283092 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:00.283196 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:00.284009 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.284788 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.286052 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.286565 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.286694 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:00.291229 (+ 4535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:55421: BlockingRecv error: recv got EOF from 127.5.56.130:55421 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 02:45:00 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:00 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:00.821826 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:00.811522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:60621)
0925 02:45:00.812024 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:00.812041 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:00.812150 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:00.812669 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:00.812672 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:00.813063 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:00.813229 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:00.813238 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.814310 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.814314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:00.815848 (+ 1534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:00.815866 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.816919 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.816926 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:00.817058 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:00.817754 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:00.817780 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:00.821533 (+ 3753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":276,"threads_started":1}
W20260925 02:45:00.822077 14738 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:00.811616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:60621 (local address 127.5.56.190:40163)
0925 02:45:00.812373 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:00.812377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:00.812393 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:45:00.812438 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:00.812441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:00.812510 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:00.812597 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:00.813370 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.814122 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.816017 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.816701 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.816834 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:00.821780 (+ 4946us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:60621: BlockingRecv error: recv got EOF from 127.5.56.131:60621 (error 108)
Metrics: {"server-negotiator.queue_time_us":668,"thread_start_us":539,"threads_started":1}
Sep 25 02:45:00 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:00 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:00.824980 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:00.815171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:56737)
0925 02:45:00.815749 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:00.815770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:00.815889 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:00.816285 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:00.816290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:00.816789 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:00.816984 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:00.817017 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.818511 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.818516 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:00.819792 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:00.819805 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.820906 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.820911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:00.821040 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:00.821662 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:00.821691 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:00.824755 (+ 3064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":301,"threads_started":1}
W20260925 02:45:00.825398 14740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:00.815259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:56737 (local address 127.5.56.190:40163)
0925 02:45:00.815991 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:00.815999 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:00.816018 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:45:00.816066 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:00.816070 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:00.816133 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:00.816213 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:00.817171 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.818306 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.819931 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:00.820687 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:00.820826 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:00.824988 (+ 4162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:56737: BlockingRecv error: recv got EOF from 127.5.56.129:56737 (error 108)
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":473,"threads_started":1}
Sep 25 02:45:01 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:01 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:01.301941 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:01.292637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:49089)
0925 02:45:01.293223 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:01.293241 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:01.293362 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:01.293692 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:01.293695 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:01.294079 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:01.294288 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:01.294304 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.295381 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.295385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:01.296336 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:01.296349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.297054 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.297058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:01.297186 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:01.297915 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:01.297935 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:01.301589 (+ 3654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":347,"threads_started":1}
W20260925 02:45:01.302196 14740 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:01.292715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:49089 (local address 127.5.56.190:40163)
0925 02:45:01.292907 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:01.292911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:01.293327 (+ 416us) server_negotiation.cc:407] Connection header received
0925 02:45:01.293477 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:01.293482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:01.293552 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:01.293645 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:01.294497 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.295233 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.296492 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.296903 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.297011 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:01.301939 (+ 4928us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:49089: BlockingRecv error: recv got EOF from 127.5.56.130:49089 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 25 02:45:01 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:01 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:01.834777 14742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:01.823028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:38657)
0925 02:45:01.823698 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:01.823718 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:01.823901 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:01.824280 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:01.824285 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:01.824716 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:01.824941 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:01.824952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.826976 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.826986 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:01.828545 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:01.828562 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.829742 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.829747 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:01.829891 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:01.830645 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:01.830670 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:01.834525 (+ 3855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":391,"threads_started":1}
W20260925 02:45:01.835103 14743 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:01.823254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:38657 (local address 127.5.56.190:40163)
0925 02:45:01.823762 (+ 508us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:01.823767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:01.823834 (+ 67us) server_negotiation.cc:407] Connection header received
0925 02:45:01.824028 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:01.824032 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:01.824111 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:01.824218 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:01.825135 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.826574 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.828702 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.829533 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.829669 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:01.834794 (+ 5125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:38657: BlockingRecv error: recv got EOF from 127.5.56.131:38657 (error 108)
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":400,"thread_start_us":283,"threads_started":1}
Sep 25 02:45:01 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:01 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:01.837852 14744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:01.825995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:46845)
0925 02:45:01.827241 (+ 1246us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:01.827259 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:01.827365 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:01.828200 (+ 835us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:01.828205 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:01.828727 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:01.828960 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:01.828972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.830484 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.830488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:01.832066 (+ 1578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:01.832094 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.833205 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.833209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:01.833375 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:01.834087 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:01.834107 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:01.837460 (+ 3353us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1117,"thread_start_us":1003,"threads_started":1}
W20260925 02:45:01.838069 14745 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:01.826004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:46845 (local address 127.5.56.190:40163)
0925 02:45:01.827876 (+ 1872us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:01.827882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:01.827901 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:45:01.827958 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:01.827962 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:01.828024 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:01.828148 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:01.829132 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.830311 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.832261 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:01.833036 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:01.833194 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:01.837750 (+ 4556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:46845: BlockingRecv error: recv got EOF from 127.5.56.129:46845 (error 108)
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":1706,"thread_start_us":1588,"threads_started":1}
Sep 25 02:45:02 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:02 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:02.313678 14746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:02.303100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:57477)
0925 02:45:02.303723 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:02.303741 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:02.303935 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:02.304436 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:02.304441 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:02.304939 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:02.305140 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:02.305150 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.306217 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.306224 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:02.307503 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:02.307518 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.308434 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.308444 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:02.308691 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:02.309623 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:02.309650 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:02.313277 (+ 3627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":309,"threads_started":1}
W20260925 02:45:02.313777 14745 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:02.303197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:57477 (local address 127.5.56.190:40163)
0925 02:45:02.303440 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:02.303449 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:02.303844 (+ 395us) server_negotiation.cc:407] Connection header received
0925 02:45:02.304200 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:02.304207 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:02.304289 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:02.304400 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:02.305301 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.306067 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.307629 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.308241 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.308376 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:02.313528 (+ 5152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:57477: BlockingRecv error: recv got EOF from 127.5.56.130:57477 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 02:45:02 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:02 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:02 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:02 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:02.852877 14749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:02.838955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:37741)
0925 02:45:02.839628 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:02.839650 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:02.839770 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:02.840153 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:02.840158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:02.840655 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:02.840861 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:02.840872 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.842309 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.842315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:02.843561 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:02.843575 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.844332 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.844336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:02.844465 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:02.848314 (+ 3849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:02.848343 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:02.852510 (+ 4167us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":317,"threads_started":1}
I20260925 02:45:02.852943 14747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:02.836091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:58451)
0925 02:45:02.836764 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:02.836790 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:02.836953 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:02.837381 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:02.837388 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:02.838023 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:02.838247 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:02.838257 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.839872 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.839881 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:02.841251 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:02.841265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.842275 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.842282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:02.842487 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:02.848318 (+ 5831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:02.848343 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:02.852586 (+ 4243us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":329,"threads_started":1}
W20260925 02:45:02.853169 14748 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:02.836231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:58451 (local address 127.5.56.190:40163)
0925 02:45:02.836977 (+ 746us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:02.836983 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:02.837052 (+ 69us) server_negotiation.cc:407] Connection header received
0925 02:45:02.837124 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:02.837128 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:02.837208 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:02.837336 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:02.838463 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.839710 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.841394 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.842079 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.842222 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:02.852868 (+ 10646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:58451: BlockingRecv error: recv got EOF from 127.5.56.131:58451 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":629,"thread_start_us":501,"threads_started":1}
W20260925 02:45:02.853448 14750 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:02.839052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:37741 (local address 127.5.56.190:40163)
0925 02:45:02.839857 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:02.839862 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:02.839884 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:45:02.839928 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:02.839931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:02.839993 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:02.840069 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:02.841023 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.842174 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.843713 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:02.844161 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:02.844263 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:02.852778 (+ 8515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:37741: BlockingRecv error: recv got EOF from 127.5.56.129:37741 (error 108)
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":516,"threads_started":1}
Sep 25 02:45:03 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:03 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:03.323873 14751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:03.314966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:59531)
0925 02:45:03.315583 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:03.315606 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:03.315753 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:03.316198 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:03.316202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:03.316587 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:03.316757 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:03.316766 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.317819 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.317823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:03.318798 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:03.318811 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.319508 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.319512 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:03.319643 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:03.320247 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:03.320266 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:03.323587 (+ 3321us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":324,"threads_started":1}
W20260925 02:45:03.324075 14750 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:03.315035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:59531 (local address 127.5.56.190:40163)
0925 02:45:03.315230 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:03.315235 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:03.315743 (+ 508us) server_negotiation.cc:407] Connection header received
0925 02:45:03.315972 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:03.315977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:03.316051 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:03.316143 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:03.316894 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.317693 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.318947 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.319357 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.319456 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:03.323858 (+ 4402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:59531: BlockingRecv error: recv got EOF from 127.5.56.130:59531 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 25 02:45:03 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:03 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:03 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:03 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:03.868610 14753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:03.854154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:54083)
0925 02:45:03.859861 (+ 5707us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:03.859884 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:03.860006 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:03.860380 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:03.860386 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:03.860883 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:03.861128 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:03.861141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.862226 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.862231 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:03.863226 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:03.863239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.864136 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.864144 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:03.864341 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:03.865070 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:03.865088 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:03.868249 (+ 3161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5371,"thread_start_us":5256,"threads_started":1}
W20260925 02:45:03.868875 14754 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:03.854185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:54083 (local address 127.5.56.190:40163)
0925 02:45:03.859038 (+ 4853us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:03.859044 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:03.860066 (+ 1022us) server_negotiation.cc:407] Connection header received
0925 02:45:03.860128 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:03.860132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:03.860194 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:03.860290 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:03.861254 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.862074 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.863384 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.863964 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.864115 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:03.868522 (+ 4407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:54083: BlockingRecv error: recv got EOF from 127.5.56.131:54083 (error 108)
Metrics: {"mutex_wait_us":115,"server-negotiator.queue_time_us":4703,"spinlock_wait_cycles":17664,"thread_start_us":4196,"threads_started":1}
I20260925 02:45:03.868947 14752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:03.854096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:57583)
0925 02:45:03.859379 (+ 5283us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:03.859411 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:03.859580 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:03.859993 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:03.859997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:03.860606 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:03.860830 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:03.860844 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.862127 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.862133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:03.863148 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:03.863161 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.864130 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.864138 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:03.864342 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:03.865153 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:03.865171 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:03.868548 (+ 3377us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5101,"thread_start_us":4913,"threads_started":1}
W20260925 02:45:03.869313 14755 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:03.854182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:57583 (local address 127.5.56.190:40163)
0925 02:45:03.858405 (+ 4223us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:03.858415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:03.859625 (+ 1210us) server_negotiation.cc:407] Connection header received
0925 02:45:03.859690 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:03.859695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:03.859801 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:03.859923 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:03.860988 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.861960 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.863320 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:03.863937 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:03.864098 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:03.868761 (+ 4663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:57583: BlockingRecv error: recv got EOF from 127.5.56.129:57583 (error 108)
Metrics: {"mutex_wait_us":86,"server-negotiator.queue_time_us":4005,"thread_start_us":4492,"threads_started":1}
Sep 25 02:45:04 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:04 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:04.337160 14756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:04.325108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:58975)
0925 02:45:04.325693 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:04.325716 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:04.325895 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:04.326424 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:04.326430 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:04.326976 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:04.327245 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:04.327259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.328846 (+ 1587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.328855 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:04.330833 (+ 1978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:04.330852 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.331880 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.331886 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:04.332080 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:04.332846 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:04.332874 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:04.336783 (+ 3909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":252,"threads_started":1}
W20260925 02:45:04.337258 14755 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:04.325352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:58975 (local address 127.5.56.190:40163)
0925 02:45:04.325542 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:04.325548 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:04.325799 (+ 251us) server_negotiation.cc:407] Connection header received
0925 02:45:04.326073 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:04.326079 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:04.326189 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:04.326321 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:04.327418 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.328516 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.330987 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.331717 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.331871 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:04.337007 (+ 5136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:58975: BlockingRecv error: recv got EOF from 127.5.56.130:58975 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 25 02:45:04 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:04 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:04 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:04 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:04.891471 14757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:04.869861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:50185)
0925 02:45:04.882262 (+ 12401us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:04.882289 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:04.882478 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:04.883186 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:04.883192 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:04.883644 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:04.883820 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:04.883829 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.884923 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.884929 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:04.885929 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:04.885941 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.886895 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.886901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:04.887100 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:04.887862 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:04.887888 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:04.891121 (+ 3233us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12255,"thread_start_us":12131,"threads_started":1}
I20260925 02:45:04.891734 14758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:04.869872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:55275)
0925 02:45:04.882659 (+ 12787us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:04.882675 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:04.882851 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:04.883311 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:04.883317 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:04.883693 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:04.883960 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:04.883972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.885120 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.885123 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:04.886147 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:04.886162 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.887168 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.887172 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:04.887311 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:04.888043 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:04.888061 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:04.891385 (+ 3324us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12653,"thread_start_us":12976,"threads_started":1}
W20260925 02:45:04.891887 14760 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:04.869961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:50185 (local address 127.5.56.190:40163)
0925 02:45:04.882827 (+ 12866us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:04.882832 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:04.882850 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:45:04.882907 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:04.882912 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:04.882990 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:04.883113 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:04.883963 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.884761 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.886085 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.886738 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.886837 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:04.891369 (+ 4532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:50185: BlockingRecv error: recv got EOF from 127.5.56.129:50185 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":12729,"thread_start_us":12530,"threads_started":1}
W20260925 02:45:04.892515 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:04.869938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:55275 (local address 127.5.56.190:40163)
0925 02:45:04.882061 (+ 12123us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:04.882071 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:04.882823 (+ 752us) server_negotiation.cc:407] Connection header received
0925 02:45:04.883017 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:04.883023 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:04.883132 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:04.883281 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:04.884122 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.884976 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.886292 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:04.886964 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:04.887085 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:04.892228 (+ 5143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:55275: BlockingRecv error: recv got EOF from 127.5.56.131:55275 (error 108)
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":11934,"thread_start_us":11832,"threads_started":1}
Sep 25 02:45:05 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:05 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:05.347700 14761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:05.338299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:42103)
0925 02:45:05.338879 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:05.338896 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:05.339014 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:05.339327 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:05.339331 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:05.339691 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:05.339904 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:05.339914 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.341009 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.341014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:05.342050 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:05.342065 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.342892 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.342898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:05.343037 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:05.343780 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:05.343802 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:05.347361 (+ 3559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":317,"threads_started":1}
W20260925 02:45:05.347716 14759 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:05.338474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:42103 (local address 127.5.56.190:40163)
0925 02:45:05.338652 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:05.338657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:05.338974 (+ 317us) server_negotiation.cc:407] Connection header received
0925 02:45:05.339122 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:05.339125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:05.339189 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:05.339274 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:05.340039 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.340824 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.342253 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.342722 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.342827 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:05.347534 (+ 4707us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:42103: BlockingRecv error: recv got EOF from 127.5.56.130:42103 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 25 02:45:05 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:05 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.131@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:05.920331 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:05.892707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.131:47993)
0925 02:45:05.908212 (+ 15505us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:05.908243 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:05.908450 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:05.909584 (+ 1134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:05.909592 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:05.910308 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:05.910676 (+ 368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:05.910692 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.912228 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.912237 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:05.913488 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:05.913500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.914467 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.914478 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:05.914701 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:05.915568 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:05.915592 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:05.919864 (+ 4272us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15314,"thread_start_us":16459,"threads_started":1}
W20260925 02:45:05.920398 14765 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:05.892793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:47993 (local address 127.5.56.190:40163)
0925 02:45:05.903786 (+ 10993us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:05.903798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:05.908602 (+ 4804us) server_negotiation.cc:407] Connection header received
0925 02:45:05.908685 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:05.908692 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:05.908834 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:05.908952 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:05.910865 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.912030 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.913639 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.914234 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.914380 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:05.920102 (+ 5722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.131:47993: BlockingRecv error: recv got EOF from 127.5.56.131:47993 (error 108)
Metrics: {"mutex_wait_us":74,"server-negotiator.queue_time_us":10794,"thread_start_us":18173,"threads_started":1}
Sep 25 02:45:05 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:05 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.129@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:05.924165 14764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:05.893348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.129:49739)
0925 02:45:05.911471 (+ 18123us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:05.911492 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:05.911628 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:05.912071 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:05.912078 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:05.912625 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:05.912905 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:05.912917 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.914795 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.914803 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:05.916295 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:05.916312 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.917890 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.917899 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:05.918154 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:05.918955 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:05.918980 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:05.923717 (+ 4737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":17979,"thread_start_us":17858,"threads_started":1}
W20260925 02:45:05.924192 14763 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:05.893717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:49739 (local address 127.5.56.190:40163)
0925 02:45:05.910954 (+ 17237us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:05.910961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:05.911603 (+ 642us) server_negotiation.cc:407] Connection header received
0925 02:45:05.911752 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:05.911759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:05.911854 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:05.911997 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:05.913090 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.914646 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.916466 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:05.917421 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:05.917578 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:05.923979 (+ 6401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.129:49739: BlockingRecv error: recv got EOF from 127.5.56.129:49739 (error 108)
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":17061,"thread_start_us":9647,"threads_started":1}
Sep 25 02:45:06 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
Sep 25 02:45:06 dist-test-slave-n563 krb5kdc[13680](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.5.56.130@KRBTEST.COM for oryx/127.5.56.190@KRBTEST.COM, Server not found in Kerberos database
I20260925 02:45:06.357542 14766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:06.349037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:40163 (local address 127.5.56.130:41207)
0925 02:45:06.349592 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:06.349610 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:06.349723 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:06.350104 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:06.350107 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:06.350594 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:06.350790 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:06.350799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:06.351814 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:06.351818 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:06.352742 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:06.352754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:06.353477 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:06.353481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:06.353592 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:06.354188 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:06.354206 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:06.357159 (+ 2953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.5.56.190:40163: Server oryx/127.5.56.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":287,"threads_started":1}
W20260925 02:45:06.357640 14763 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:06.349100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:41207 (local address 127.5.56.190:40163)
0925 02:45:06.349286 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:06.349290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:06.349702 (+ 412us) server_negotiation.cc:407] Connection header received
0925 02:45:06.349887 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:06.349893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:06.349972 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:06.350061 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:06.350937 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:06.351690 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:06.352913 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:06.353340 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:06.353436 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:06.357399 (+ 3963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.5.56.130:41207: BlockingRecv error: recv got EOF from 127.5.56.130:41207 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:45:06.683584 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 14275
I20260925 02:45:06.697148 14768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:06.684889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60526 (local address 127.5.56.129:37205)
0925 02:45:06.685504 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:06.685512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:06.685541 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:45:06.685605 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:06.685612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:06.685838 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:06.686027 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:06.687767 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:06.688887 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:06.690235 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:06.690934 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:06.691047 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:06.693705 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:06.693715 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:06.693730 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:45:06.693768 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:06.696620 (+ 2852us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.56.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 02:45:06.696912 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60526: Request ticket server kudu/127.5.56.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":312,"threads_started":1}
W20260925 02:45:06.697284 14768 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60526: Request ticket server kudu/127.5.56.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 02:45:06.697410 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 14275: Not authorized: Client connection negotiation failed: client connection to 127.5.56.129:37205: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.56.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 02:45:06.697656 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 14275
I20260925 02:45:06.716058 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 14426
I20260925 02:45:06.729012 14769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:06.717197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47708 (local address 127.5.56.130:38813)
0925 02:45:06.717802 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:06.717810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:06.717841 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:45:06.717929 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:06.717939 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:06.718164 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:06.718372 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:06.719785 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:06.720827 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:06.722104 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:06.722812 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:06.722959 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:06.725645 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:06.725655 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:06.725670 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:45:06.725711 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:06.728248 (+ 2537us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.56.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 02:45:06.728631 (+ 383us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47708: Request ticket server kudu/127.5.56.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":301,"threads_started":1}
W20260925 02:45:06.729138 14769 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47708: Request ticket server kudu/127.5.56.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 02:45:06.729104 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 14426: Not authorized: Client connection negotiation failed: client connection to 127.5.56.130:38813: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.56.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 02:45:06.729295 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 14426
I20260925 02:45:06.747953 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 14560
I20260925 02:45:06.761159 14770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:06.748907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40802 (local address 127.5.56.131:35273)
0925 02:45:06.749523 (+ 616us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:06.749531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:06.749558 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:45:06.749620 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:06.749627 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:06.749861 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:06.750026 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:06.751593 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:06.752628 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:06.754008 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:06.754785 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:06.754894 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:06.757797 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:06.757807 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:06.757822 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:45:06.757859 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:06.760569 (+ 2710us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.56.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0925 02:45:06.760877 (+ 308us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40802: Request ticket server kudu/127.5.56.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":304,"threads_started":1}
W20260925 02:45:06.761307 14770 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40802: Request ticket server kudu/127.5.56.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260925 02:45:06.761410 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 14560: Not authorized: Client connection negotiation failed: client connection to 127.5.56.131:35273: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.5.56.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260925 02:45:06.761632 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 14560
I20260925 02:45:06.779429 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 14205
I20260925 02:45:06.780877 14268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:06.903867 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 14205
2026-09-25T02:45:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21507 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T02:45:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:45:06Z Disabled control of system clock
I20260925 02:45:06.987300 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:33311
--webserver_interface=127.5.56.190
--webserver_port=0
--builtin_ntp_servers=127.5.56.148:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:33311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 02:45:07.344002 14775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:07.344342 14775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:07.344506 14775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:07.352217 14775 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:45:07.352342 14775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:07.352445 14775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:07.352527 14775 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:45:07.352572 14775 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:45:07.362296 14775 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:33311
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:33311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:07.363750 14775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:07.365803 14775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:07.379079 14780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:07.379056 14781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:07.379014 14783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:07.379980 14775 server_base.cc:1034] running on GCE node
I20260925 02:45:07.380895 14775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:07.382575 14775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:07.383832 14775 hybrid_clock.cc:648] HybridClock initialized: now 1790304307383777 us; error 50 us; skew 500 ppm
I20260925 02:45:07.384217 14775 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:45:07.386855 14775 webserver.cc:460] Webserver started at http://127.5.56.190:34845/ using document root <none> and password file <none>
I20260925 02:45:07.387647 14775 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:07.387748 14775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:07.388058 14775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:07.391464 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "c16ddfdad98d4bf2a6a1b9f4451c7cd7"
format_stamp: "Formatted at 2026-09-25 02:45:07 on dist-test-slave-n563"
server_key: "e4f66b89e85787a64f013f50e063eb4c"
server_key_iv: "50ed231a5670f15e4f9dbe64efb7e80c"
server_key_version: "encryptionkey@0"
I20260925 02:45:07.392341 14775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "c16ddfdad98d4bf2a6a1b9f4451c7cd7"
format_stamp: "Formatted at 2026-09-25 02:45:07 on dist-test-slave-n563"
server_key: "e4f66b89e85787a64f013f50e063eb4c"
server_key_iv: "50ed231a5670f15e4f9dbe64efb7e80c"
server_key_version: "encryptionkey@0"
I20260925 02:45:07.398373 14775 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260925 02:45:07.402722 14789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:07.404376 14775 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:45:07.404618 14775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "c16ddfdad98d4bf2a6a1b9f4451c7cd7"
format_stamp: "Formatted at 2026-09-25 02:45:07 on dist-test-slave-n563"
server_key: "e4f66b89e85787a64f013f50e063eb4c"
server_key_iv: "50ed231a5670f15e4f9dbe64efb7e80c"
server_key_version: "encryptionkey@0"
I20260925 02:45:07.404896 14775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:07.417150 14775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:07.418159 14775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:07.418548 14775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:07.439225 14775 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:33311
I20260925 02:45:07.439244 14841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:33311 every 8 connection(s)
I20260925 02:45:07.440807 14775 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:45:07.442837 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 14775
I20260925 02:45:07.443069 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:45:07.443965 5346 external_mini_cluster.cc:1428] Setting key cedc41a3c27dad8c652b157aca49c166
I20260925 02:45:07.446285 14842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:45:07.460151 14845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:07.446327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57140 (local address 127.5.56.190:33311)
0925 02:45:07.447511 (+ 1184us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:07.447523 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:07.447571 (+ 48us) server_negotiation.cc:407] Connection header received
0925 02:45:07.449252 (+ 1681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:07.449279 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:07.449860 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:07.450282 (+ 422us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:07.451157 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:07.452701 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:07.454837 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:07.455600 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:07.455765 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:07.457268 (+ 1503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:07.457280 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:07.457298 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:07.457958 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:07.458086 (+ 128us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:07.458115 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:07.458223 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:07.458586 (+ 363us) server_negotiation.cc:299] Negotiation successful
0925 02:45:07.459034 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":394,"threads_started":1}
I20260925 02:45:07.460892 14842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7: Bootstrap starting.
I20260925 02:45:07.464780 14842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7: Neither blocks nor log segments found. Creating new log.
I20260925 02:45:07.465993 14842 log.cc:824] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:45:07.469552 14842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7: No bootstrap required, opened a new log
I20260925 02:45:07.477444 14842 raft_consensus.cc:348] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ddfdad98d4bf2a6a1b9f4451c7cd7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33311 } }
I20260925 02:45:07.477928 14842 raft_consensus.cc:374] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:45:07.478052 14842 raft_consensus.cc:729] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c16ddfdad98d4bf2a6a1b9f4451c7cd7, State: Initialized, Role: FOLLOWER
I20260925 02:45:07.478825 14842 consensus_queue.cc:260] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ddfdad98d4bf2a6a1b9f4451c7cd7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33311 } }
I20260925 02:45:07.479125 14842 raft_consensus.cc:388] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:45:07.479290 14842 raft_consensus.cc:482] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:45:07.479485 14842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:45:07.481846 14842 raft_consensus.cc:504] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ddfdad98d4bf2a6a1b9f4451c7cd7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33311 } }
I20260925 02:45:07.482553 14842 leader_election.cc:302] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c16ddfdad98d4bf2a6a1b9f4451c7cd7; no voters:
I20260925 02:45:07.483284 14842 leader_election.cc:288] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:45:07.483484 14847 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:45:07.484953 14847 raft_consensus.cc:686] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [term 1 LEADER]: Becoming Leader. State: Replica: c16ddfdad98d4bf2a6a1b9f4451c7cd7, State: Running, Role: LEADER
I20260925 02:45:07.485622 14847 consensus_queue.cc:237] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ddfdad98d4bf2a6a1b9f4451c7cd7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33311 } }
I20260925 02:45:07.486022 14842 sys_catalog.cc:564] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:45:07.488897 14849 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c16ddfdad98d4bf2a6a1b9f4451c7cd7. Latest consensus state: current_term: 1 leader_uuid: "c16ddfdad98d4bf2a6a1b9f4451c7cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ddfdad98d4bf2a6a1b9f4451c7cd7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33311 } } }
I20260925 02:45:07.489545 14849 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:45:07.488996 14848 sys_catalog.cc:455] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c16ddfdad98d4bf2a6a1b9f4451c7cd7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c16ddfdad98d4bf2a6a1b9f4451c7cd7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33311 } } }
I20260925 02:45:07.491439 14848 sys_catalog.cc:458] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:45:07.492547 14856 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:45:07.497184 14856 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:45:07.507334 14856 catalog_manager.cc:1347] Generated new cluster ID: 7d36d9cc431f4b7f9b022fa6d22e40be
I20260925 02:45:07.507475 14856 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:45:07.541622 14856 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:45:07.543681 14856 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:45:07.554111 14856 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c16ddfdad98d4bf2a6a1b9f4451c7cd7: Generated new TSK 0
I20260925 02:45:07.555436 14856 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:45:07.571683 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33311
--builtin_ntp_servers=127.5.56.148:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 02:45:07.943846 14866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:07.944173 14866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:07.944263 14866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:07.952584 14866 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:07.952744 14866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:07.952875 14866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:45:07.963533 14866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:07.964973 14866 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:07.967135 14866 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:07.982187 14871 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:07.982091 14874 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:07.982165 14872 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:07.983184 14866 server_base.cc:1034] running on GCE node
I20260925 02:45:07.983878 14866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:07.985014 14866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:07.986225 14866 hybrid_clock.cc:648] HybridClock initialized: now 1790304307986178 us; error 47 us; skew 500 ppm
I20260925 02:45:07.986683 14866 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:45:07.993275 14866 webserver.cc:460] Webserver started at http://127.5.56.129:36731/ using document root <none> and password file <none>
I20260925 02:45:07.994179 14866 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:07.994319 14866 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:07.994748 14866 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:07.997884 14866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "ef93cb98a6a64b658d672e2a6060db45"
format_stamp: "Formatted at 2026-09-25 02:45:07 on dist-test-slave-n563"
server_key: "1d351f4af2a7b037a74ebaac1bed1d8f"
server_key_iv: "a22b90bcade411e471fda2d4b9fd2fd3"
server_key_version: "encryptionkey@0"
I20260925 02:45:07.998819 14866 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "ef93cb98a6a64b658d672e2a6060db45"
format_stamp: "Formatted at 2026-09-25 02:45:07 on dist-test-slave-n563"
server_key: "1d351f4af2a7b037a74ebaac1bed1d8f"
server_key_iv: "a22b90bcade411e471fda2d4b9fd2fd3"
server_key_version: "encryptionkey@0"
I20260925 02:45:08.004688 14866 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:45:08.008829 14880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:08.011052 14866 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260925 02:45:08.011273 14866 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "ef93cb98a6a64b658d672e2a6060db45"
format_stamp: "Formatted at 2026-09-25 02:45:07 on dist-test-slave-n563"
server_key: "1d351f4af2a7b037a74ebaac1bed1d8f"
server_key_iv: "a22b90bcade411e471fda2d4b9fd2fd3"
server_key_version: "encryptionkey@0"
I20260925 02:45:08.011533 14866 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:08.047410 14866 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:08.048478 14866 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:08.048827 14866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:08.049907 14866 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:45:08.051558 14866 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:45:08.051651 14866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:08.051793 14866 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:45:08.051891 14866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:08.087630 14866 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:42545
I20260925 02:45:08.087692 14993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:42545 every 8 connection(s)
I20260925 02:45:08.089094 14866 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:45:08.089562 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 14866
I20260925 02:45:08.089763 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:45:08.090739 5346 external_mini_cluster.cc:1428] Setting key 371f3560d88d9a1d8d64908631c737a5
I20260925 02:45:08.093961 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33311
--builtin_ntp_servers=127.5.56.148:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 02:45:08.104252 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:08.092209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:55317 (local address 127.5.56.190:33311)
0925 02:45:08.092710 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:08.092717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:08.094432 (+ 1715us) server_negotiation.cc:407] Connection header received
0925 02:45:08.096313 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:08.096321 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:08.096429 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:08.096558 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:08.097729 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:08.098938 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:08.100632 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:08.101069 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:08.101177 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:08.103057 (+ 1880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:08.103067 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:08.103075 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:08.103107 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:08.103166 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:08.103186 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:08.103294 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:08.103584 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 02:45:08.103716 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":243,"threads_started":1}
I20260925 02:45:08.105182 14997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:08.093070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:33311 (local address 127.5.56.129:55317)
0925 02:45:08.094224 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:08.094274 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:08.095994 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:08.096781 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:08.096790 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:08.096888 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:08.097522 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:08.097540 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:08.099104 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:08.099107 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:08.100419 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:08.100432 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:08.101352 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:08.101361 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:08.101728 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:08.102669 (+ 941us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:45:08.102672 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:08.102719 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:45:08.102822 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:45:08.103337 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:45:08.103348 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:45:08.103356 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:45:08.103536 (+ 180us) client_negotiation.cc:240] Negotiation successful
0925 02:45:08.103893 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":365,"threads_started":1}
I20260925 02:45:08.107059 14994 heartbeater.cc:347] Connected to a master server at 127.5.56.190:33311
I20260925 02:45:08.107425 14994 heartbeater.cc:464] Registering TS with master...
I20260925 02:45:08.108302 14994 heartbeater.cc:511] Master 127.5.56.190:33311 requested a full tablet report, sending...
I20260925 02:45:08.110847 14806 ts_manager.cc:194] Registered new tserver with Master: ef93cb98a6a64b658d672e2a6060db45 (127.5.56.129:42545)
I20260925 02:45:08.112406 14806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.129:55317
W20260925 02:45:08.463881 14998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:08.464190 14998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:08.464329 14998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:08.472137 14998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:08.472267 14998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:08.472437 14998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:45:08.482051 14998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:08.483521 14998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:08.485493 14998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:08.501137 15003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:08.501076 15004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:08.501060 15006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:08.502712 14998 server_base.cc:1034] running on GCE node
I20260925 02:45:08.503479 14998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:08.504645 14998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:08.505865 14998 hybrid_clock.cc:648] HybridClock initialized: now 1790304308505818 us; error 51 us; skew 500 ppm
I20260925 02:45:08.506235 14998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:45:08.509183 14998 webserver.cc:460] Webserver started at http://127.5.56.130:40659/ using document root <none> and password file <none>
I20260925 02:45:08.510047 14998 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:08.510157 14998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:08.510689 14998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:08.514019 14998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "30c0fc76de9540f686e953cd806a1bd8"
format_stamp: "Formatted at 2026-09-25 02:45:08 on dist-test-slave-n563"
server_key: "b681a7bc3a772d2c47ed71a910aaa0e4"
server_key_iv: "59279988a315fb12777f03cef8de5da9"
server_key_version: "encryptionkey@0"
I20260925 02:45:08.515004 14998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "30c0fc76de9540f686e953cd806a1bd8"
format_stamp: "Formatted at 2026-09-25 02:45:08 on dist-test-slave-n563"
server_key: "b681a7bc3a772d2c47ed71a910aaa0e4"
server_key_iv: "59279988a315fb12777f03cef8de5da9"
server_key_version: "encryptionkey@0"
I20260925 02:45:08.521144 14998 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260925 02:45:08.525367 15012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:08.527073 14998 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:45:08.527304 14998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "30c0fc76de9540f686e953cd806a1bd8"
format_stamp: "Formatted at 2026-09-25 02:45:08 on dist-test-slave-n563"
server_key: "b681a7bc3a772d2c47ed71a910aaa0e4"
server_key_iv: "59279988a315fb12777f03cef8de5da9"
server_key_version: "encryptionkey@0"
I20260925 02:45:08.527591 14998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:08.546020 14998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:08.547112 14998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:08.547485 14998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:08.548610 14998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:45:08.550324 14998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:45:08.550495 14998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 02:45:08.550640 14998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:45:08.550734 14998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:08.586627 14998 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:45887
I20260925 02:45:08.586665 15125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:45887 every 8 connection(s)
I20260925 02:45:08.588208 14998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:45:08.592677 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 14998
I20260925 02:45:08.592859 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:45:08.593432 5346 external_mini_cluster.cc:1428] Setting key 9cab8d96105d07066dc75b833a808ace
I20260925 02:45:08.596541 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33311
--builtin_ntp_servers=127.5.56.148:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260925 02:45:08.602782 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:08.591311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:53907 (local address 127.5.56.190:33311)
0925 02:45:08.591548 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:08.591555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:08.593415 (+ 1860us) server_negotiation.cc:407] Connection header received
0925 02:45:08.595703 (+ 2288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:08.595709 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:08.595801 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:08.595917 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:08.597229 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:08.598005 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:08.599785 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:08.600217 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:08.600366 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:08.601881 (+ 1515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:08.601886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:08.601890 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:08.601912 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:08.601948 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:08.601960 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:08.602059 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:08.602305 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 02:45:08.602454 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:45:08.604192 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:08.591996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:33311 (local address 127.5.56.130:53907)
0925 02:45:08.593220 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:08.593269 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:08.595393 (+ 2124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:08.596198 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:08.596208 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:08.596317 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:08.597044 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:08.597064 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:08.598188 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:08.598193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:08.599614 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:08.599628 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:08.600474 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:08.600482 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:08.600735 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:08.601514 (+ 779us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:45:08.601516 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:08.601566 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:45:08.601658 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:45:08.602059 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:45:08.602074 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:45:08.602086 (+ 12us) client_negotiation.cc:769] Sending connection context
0925 02:45:08.602295 (+ 209us) client_negotiation.cc:240] Negotiation successful
0925 02:45:08.602792 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":421,"threads_started":1}
I20260925 02:45:08.605973 15126 heartbeater.cc:347] Connected to a master server at 127.5.56.190:33311
I20260925 02:45:08.606324 15126 heartbeater.cc:464] Registering TS with master...
I20260925 02:45:08.607251 15126 heartbeater.cc:511] Master 127.5.56.190:33311 requested a full tablet report, sending...
I20260925 02:45:08.608913 14806 ts_manager.cc:194] Registered new tserver with Master: 30c0fc76de9540f686e953cd806a1bd8 (127.5.56.130:45887)
I20260925 02:45:08.609822 14806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.130:53907
W20260925 02:45:08.959789 15129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:08.960177 15129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:08.960345 15129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:08.968063 15129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:08.968261 15129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:08.968451 15129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:45:08.978240 15129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:08.979781 15129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:08.981732 15129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:08.996484 15134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:08.996512 15137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:08.996608 15135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:08.997169 15129 server_base.cc:1034] running on GCE node
I20260925 02:45:08.997855 15129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:08.999028 15129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:09.000241 15129 hybrid_clock.cc:648] HybridClock initialized: now 1790304309000197 us; error 46 us; skew 500 ppm
I20260925 02:45:09.000610 15129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:45:09.003237 15129 webserver.cc:460] Webserver started at http://127.5.56.131:45187/ using document root <none> and password file <none>
I20260925 02:45:09.004021 15129 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:09.004143 15129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:09.004515 15129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:09.007687 15129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "ddf4a84ab00e4f588de69cc4e166db2b"
format_stamp: "Formatted at 2026-09-25 02:45:09 on dist-test-slave-n563"
server_key: "12e2283767285dd2f584267728cabb8a"
server_key_iv: "2fcc4d8f259035eaacac7e946b9153b3"
server_key_version: "encryptionkey@0"
I20260925 02:45:09.008538 15129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "ddf4a84ab00e4f588de69cc4e166db2b"
format_stamp: "Formatted at 2026-09-25 02:45:09 on dist-test-slave-n563"
server_key: "12e2283767285dd2f584267728cabb8a"
server_key_iv: "2fcc4d8f259035eaacac7e946b9153b3"
server_key_version: "encryptionkey@0"
I20260925 02:45:09.014767 15129 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260925 02:45:09.018771 15143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:09.020465 15129 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:45:09.020682 15129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "ddf4a84ab00e4f588de69cc4e166db2b"
format_stamp: "Formatted at 2026-09-25 02:45:09 on dist-test-slave-n563"
server_key: "12e2283767285dd2f584267728cabb8a"
server_key_iv: "2fcc4d8f259035eaacac7e946b9153b3"
server_key_version: "encryptionkey@0"
I20260925 02:45:09.020925 15129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:09.047240 15129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:09.048251 15129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:09.048606 15129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:09.049746 15129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:45:09.051457 15129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:45:09.051580 15129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:09.051723 15129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:45:09.051811 15129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:09.087181 15129 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:37507
I20260925 02:45:09.087400 15256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:37507 every 8 connection(s)
I20260925 02:45:09.088953 15129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:45:09.094307 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 15129
I20260925 02:45:09.094512 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:45:09.095109 5346 external_mini_cluster.cc:1428] Setting key 38c8021d4d0277f8dfae0c5d02e091a0
I20260925 02:45:09.103610 14996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:09.092005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:49927 (local address 127.5.56.190:33311)
0925 02:45:09.092194 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:09.092199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:09.093776 (+ 1577us) server_negotiation.cc:407] Connection header received
0925 02:45:09.096347 (+ 2571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:09.096351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:09.096451 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:09.096559 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:09.098257 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:09.099104 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:09.100791 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:09.101199 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:09.101300 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:09.102858 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:09.102864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:09.102867 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:09.102888 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:09.102923 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:09.102935 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:09.103017 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:09.103239 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 02:45:09.103331 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:45:09.104838 15259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:09.092575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:33311 (local address 127.5.56.131:49927)
0925 02:45:09.093594 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:09.093630 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:09.096023 (+ 2393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:09.097035 (+ 1012us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:09.097051 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:09.097186 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:09.098039 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:09.098068 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:09.099261 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:09.099265 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:09.100646 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:09.100659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:09.101440 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:09.101446 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:09.101699 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:09.102496 (+ 797us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:45:09.102499 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:09.102554 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:45:09.102651 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:45:09.103035 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:45:09.103043 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:45:09.103049 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:45:09.103208 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 02:45:09.103554 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":347,"threads_started":1}
I20260925 02:45:09.106360 15257 heartbeater.cc:347] Connected to a master server at 127.5.56.190:33311
I20260925 02:45:09.106702 15257 heartbeater.cc:464] Registering TS with master...
I20260925 02:45:09.107491 15257 heartbeater.cc:511] Master 127.5.56.190:33311 requested a full tablet report, sending...
I20260925 02:45:09.109093 14806 ts_manager.cc:194] Registered new tserver with Master: ddf4a84ab00e4f588de69cc4e166db2b (127.5.56.131:37507)
I20260925 02:45:09.110179 14806 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.131:49927
I20260925 02:45:09.111380 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:45:09.116333 14994 heartbeater.cc:503] Master 127.5.56.190:33311 was elected leader, sending a full tablet report...
W20260925 02:45:09.119330 14996 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:09.117383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57146 (local address 127.5.56.190:33311)
0925 02:45:09.117604 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:09.117610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:09.118150 (+ 540us) server_negotiation.cc:407] Connection header received
0925 02:45:09.118363 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:09.118369 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:09.118477 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:09.118591 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:09.119076 (+ 485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57146: BlockingRecv error: recv got EOF from 127.0.0.1:57146 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:45:09.121026 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 14866
I20260925 02:45:09.129734 15266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:09.122072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43580 (local address 127.5.56.129:42545)
0925 02:45:09.122719 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:09.122727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:09.122753 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:45:09.122825 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:09.122831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:09.123061 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:09.123250 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:09.123716 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:09.124744 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:09.126088 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:09.126751 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:09.126900 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:09.128127 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:09.128137 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:09.128153 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:09.128762 (+ 609us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:09.128818 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:09.128846 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:09.128986 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:09.129183 (+ 197us) server_negotiation.cc:299] Negotiation successful
0925 02:45:09.129342 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":326,"threads_started":1}
I20260925 02:45:09.131148 14989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:09.216167 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 14866
I20260925 02:45:09.236513 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 14998
I20260925 02:45:09.244350 15268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:09.237378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53000 (local address 127.5.56.130:45887)
0925 02:45:09.237967 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:09.237975 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:09.238005 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:45:09.238077 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:09.238084 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:09.238309 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:09.238541 (+ 232us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:09.238843 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:09.239875 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:09.241092 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:09.241683 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:09.241785 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:09.242920 (+ 1135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:09.242928 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:09.242943 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:09.243579 (+ 636us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:09.243637 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:09.243657 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:09.243750 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:09.243891 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 02:45:09.244032 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":317,"threads_started":1}
I20260925 02:45:09.245752 15121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:09.329442 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 14998
I20260925 02:45:09.346599 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 15129
I20260925 02:45:09.354938 15270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:09.347673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.5.56.131:37507)
0925 02:45:09.348346 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:09.348353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:09.348387 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:45:09.348449 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:09.348456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:09.348699 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:09.348825 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:09.349247 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:09.350317 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:09.351637 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:09.352225 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:09.352328 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:09.353419 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:09.353428 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:09.353443 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:09.354092 (+ 649us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:09.354143 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:09.354164 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:09.354262 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:09.354493 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 02:45:09.354615 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":363,"threads_started":1}
I20260925 02:45:09.356284 15252 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:09.443904 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 15129
I20260925 02:45:09.463174 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 14775
I20260925 02:45:09.464576 14837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:09.572163 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 14775
2026-09-25T02:45:09Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2662 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T02:45:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:45:09Z Disabled control of system clock
I20260925 02:45:09.650504 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:36631
--webserver_interface=127.5.56.190
--webserver_port=0
--builtin_ntp_servers=127.5.56.148:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:36631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:45:09.998603 15276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:09.998934 15276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:09.999122 15276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:10.006819 15276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:45:10.006950 15276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:10.007055 15276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:10.007166 15276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:45:10.007233 15276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:45:10.017185 15276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:36631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:36631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:10.018652 15276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:10.020633 15276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:10.033139 15285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:10.033175 15283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:10.033103 15282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:10.034149 15276 server_base.cc:1034] running on GCE node
I20260925 02:45:10.035109 15276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:10.036553 15276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:10.037765 15276 hybrid_clock.cc:648] HybridClock initialized: now 1790304310037719 us; error 34 us; skew 500 ppm
I20260925 02:45:10.038234 15276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:45:10.040762 15276 webserver.cc:460] Webserver started at http://127.5.56.190:35353/ using document root <none> and password file <none>
I20260925 02:45:10.041529 15276 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:10.041649 15276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:10.042013 15276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:10.047051 15276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "b5b369aeae0a4b70bc12acb804f2e32c"
format_stamp: "Formatted at 2026-09-25 02:45:10 on dist-test-slave-n563"
server_key: "322dd41277d2956619d8935b4ce28271"
server_key_iv: "c5c4720ec9964c0a6fbc52f3faddf875"
server_key_version: "encryptionkey@0"
I20260925 02:45:10.047926 15276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "b5b369aeae0a4b70bc12acb804f2e32c"
format_stamp: "Formatted at 2026-09-25 02:45:10 on dist-test-slave-n563"
server_key: "322dd41277d2956619d8935b4ce28271"
server_key_iv: "c5c4720ec9964c0a6fbc52f3faddf875"
server_key_version: "encryptionkey@0"
I20260925 02:45:10.053676 15276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:45:10.057549 15291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:10.059223 15276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:45:10.059468 15276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "b5b369aeae0a4b70bc12acb804f2e32c"
format_stamp: "Formatted at 2026-09-25 02:45:10 on dist-test-slave-n563"
server_key: "322dd41277d2956619d8935b4ce28271"
server_key_iv: "c5c4720ec9964c0a6fbc52f3faddf875"
server_key_version: "encryptionkey@0"
I20260925 02:45:10.059748 15276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:10.066468 15276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:10.067418 15276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:10.067773 15276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:10.088730 15276 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:36631
I20260925 02:45:10.088749 15343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:36631 every 8 connection(s)
I20260925 02:45:10.090405 15276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:45:10.094903 15344 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:45:10.095809 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 15276
I20260925 02:45:10.096057 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:45:10.096894 5346 external_mini_cluster.cc:1428] Setting key 1807fe385df8bf4c33f2b97166c8a85b
I20260925 02:45:10.106205 15347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:10.099243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42988 (local address 127.5.56.190:36631)
0925 02:45:10.100424 (+ 1181us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:10.100436 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:10.100485 (+ 49us) server_negotiation.cc:407] Connection header received
0925 02:45:10.102210 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:10.102236 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:10.102824 (+ 588us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:10.103267 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:10.104434 (+ 1167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:10.104440 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:10.104453 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:10.104469 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:10.104545 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:10.104561 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:10.104815 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:10.104955 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 02:45:10.105218 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":700,"thread_start_us":440,"threads_started":1}
I20260925 02:45:10.109825 15344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c: Bootstrap starting.
I20260925 02:45:10.113241 15344 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c: Neither blocks nor log segments found. Creating new log.
I20260925 02:45:10.114319 15344 log.cc:824] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:45:10.117602 15344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c: No bootstrap required, opened a new log
I20260925 02:45:10.125411 15344 raft_consensus.cc:348] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b369aeae0a4b70bc12acb804f2e32c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36631 } }
I20260925 02:45:10.125874 15344 raft_consensus.cc:374] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:45:10.125993 15344 raft_consensus.cc:729] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5b369aeae0a4b70bc12acb804f2e32c, State: Initialized, Role: FOLLOWER
I20260925 02:45:10.126710 15344 consensus_queue.cc:260] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b369aeae0a4b70bc12acb804f2e32c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36631 } }
I20260925 02:45:10.126962 15344 raft_consensus.cc:388] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:45:10.127125 15344 raft_consensus.cc:482] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:45:10.127297 15344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:45:10.129946 15344 raft_consensus.cc:504] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b369aeae0a4b70bc12acb804f2e32c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36631 } }
I20260925 02:45:10.130577 15344 leader_election.cc:302] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b5b369aeae0a4b70bc12acb804f2e32c; no voters:
I20260925 02:45:10.131213 15344 leader_election.cc:288] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:45:10.131381 15349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:45:10.132119 15349 raft_consensus.cc:686] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [term 1 LEADER]: Becoming Leader. State: Replica: b5b369aeae0a4b70bc12acb804f2e32c, State: Running, Role: LEADER
I20260925 02:45:10.132805 15349 consensus_queue.cc:237] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b369aeae0a4b70bc12acb804f2e32c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36631 } }
I20260925 02:45:10.133441 15344 sys_catalog.cc:564] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:45:10.135805 15351 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b5b369aeae0a4b70bc12acb804f2e32c. Latest consensus state: current_term: 1 leader_uuid: "b5b369aeae0a4b70bc12acb804f2e32c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b369aeae0a4b70bc12acb804f2e32c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36631 } } }
I20260925 02:45:10.136296 15351 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [sys.catalog]: This master's current role is: LEADER
I20260925 02:45:10.136204 15350 sys_catalog.cc:455] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b5b369aeae0a4b70bc12acb804f2e32c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b5b369aeae0a4b70bc12acb804f2e32c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36631 } } }
I20260925 02:45:10.136535 15350 sys_catalog.cc:458] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c [sys.catalog]: This master's current role is: LEADER
I20260925 02:45:10.138001 15354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:45:10.144160 15354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:45:10.153919 15354 catalog_manager.cc:1347] Generated new cluster ID: 8ec73d0fccf64340803a55661f76da11
I20260925 02:45:10.154060 15354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:45:10.169098 15354 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:45:10.169621 15354 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:45:10.187387 15354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b5b369aeae0a4b70bc12acb804f2e32c: Generated new TSK 0
I20260925 02:45:10.188388 15354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:45:10.195840 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36631
--builtin_ntp_servers=127.5.56.148:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 02:45:10.568476 15368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:10.568768 15368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:10.568938 15368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:10.576843 15368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:10.576957 15368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:10.577128 15368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:45:10.586884 15368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:10.588251 15368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:10.590121 15368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:10.603832 15373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:10.603868 15374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:10.603809 15377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:10.604671 15368 server_base.cc:1034] running on GCE node
I20260925 02:45:10.605322 15368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:10.606534 15368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:10.607769 15368 hybrid_clock.cc:648] HybridClock initialized: now 1790304310607716 us; error 49 us; skew 500 ppm
I20260925 02:45:10.608151 15368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:45:10.610613 15368 webserver.cc:460] Webserver started at http://127.5.56.129:42759/ using document root <none> and password file <none>
I20260925 02:45:10.611344 15368 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:10.611478 15368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:10.611835 15368 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:10.614945 15368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "32ffe1d613c4446ba74044e3148b795b"
format_stamp: "Formatted at 2026-09-25 02:45:10 on dist-test-slave-n563"
server_key: "0187fce0dda653c5a5875c28cf8c9411"
server_key_iv: "6a93dc7784d4475ebd41a50b74e67593"
server_key_version: "encryptionkey@0"
I20260925 02:45:10.615828 15368 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "32ffe1d613c4446ba74044e3148b795b"
format_stamp: "Formatted at 2026-09-25 02:45:10 on dist-test-slave-n563"
server_key: "0187fce0dda653c5a5875c28cf8c9411"
server_key_iv: "6a93dc7784d4475ebd41a50b74e67593"
server_key_version: "encryptionkey@0"
I20260925 02:45:10.621241 15368 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:45:10.624881 15383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:10.626554 15368 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:45:10.626729 15368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "32ffe1d613c4446ba74044e3148b795b"
format_stamp: "Formatted at 2026-09-25 02:45:10 on dist-test-slave-n563"
server_key: "0187fce0dda653c5a5875c28cf8c9411"
server_key_iv: "6a93dc7784d4475ebd41a50b74e67593"
server_key_version: "encryptionkey@0"
I20260925 02:45:10.626981 15368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:10.633034 15368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:10.633991 15368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:10.634275 15368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:10.635361 15368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:45:10.636946 15368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:45:10.637074 15368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:10.637238 15368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:45:10.637321 15368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:10.672113 15368 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:40921
I20260925 02:45:10.672170 15496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:40921 every 8 connection(s)
I20260925 02:45:10.673511 15368 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:45:10.677361 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 15368
I20260925 02:45:10.677601 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:45:10.678313 5346 external_mini_cluster.cc:1428] Setting key 2badd6caf78c79ef8fad7602e5a6be3b
I20260925 02:45:10.681159 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36631
--builtin_ntp_servers=127.5.56.148:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 02:45:10.682698 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:10.676301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:51939 (local address 127.5.56.190:36631)
0925 02:45:10.676801 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:10.676807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:10.678240 (+ 1433us) server_negotiation.cc:407] Connection header received
0925 02:45:10.680237 (+ 1997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:10.680242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:10.680307 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:10.680438 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:10.681849 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:10.681855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:10.681860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:10.681879 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:10.681931 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:10.681943 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:10.682081 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:10.682281 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 02:45:10.682353 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":257,"threads_started":1}
I20260925 02:45:10.683676 15500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:10.676941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36631 (local address 127.5.56.129:51939)
0925 02:45:10.678045 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:10.678100 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:10.679918 (+ 1818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:10.680694 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:10.680703 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:10.680805 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:10.681542 (+ 737us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:45:10.681545 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:10.681584 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:45:10.681673 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:45:10.682047 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:45:10.682061 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:45:10.682069 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:45:10.682368 (+ 299us) client_negotiation.cc:240] Negotiation successful
0925 02:45:10.682677 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":364,"threads_started":1}
I20260925 02:45:10.685343 15497 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36631
I20260925 02:45:10.685653 15497 heartbeater.cc:464] Registering TS with master...
I20260925 02:45:10.686435 15497 heartbeater.cc:511] Master 127.5.56.190:36631 requested a full tablet report, sending...
I20260925 02:45:10.688827 15309 ts_manager.cc:194] Registered new tserver with Master: 32ffe1d613c4446ba74044e3148b795b (127.5.56.129:40921)
W20260925 02:45:11.024451 15501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:11.024832 15501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:11.025004 15501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:11.032977 15501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:11.033133 15501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:11.033317 15501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:45:11.043098 15501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:11.044523 15501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:11.046517 15501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:11.059976 15509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:11.060032 15507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:11.060055 15506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:11.063953 15501 server_base.cc:1034] running on GCE node
I20260925 02:45:11.064680 15501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:11.065793 15501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:11.067026 15501 hybrid_clock.cc:648] HybridClock initialized: now 1790304311066984 us; error 39 us; skew 500 ppm
I20260925 02:45:11.067374 15501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:45:11.069751 15501 webserver.cc:460] Webserver started at http://127.5.56.130:37919/ using document root <none> and password file <none>
I20260925 02:45:11.070559 15501 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:11.070694 15501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:11.071051 15501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:11.074003 15501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "f325afc3ad714f93bd2f023035dedd23"
format_stamp: "Formatted at 2026-09-25 02:45:11 on dist-test-slave-n563"
server_key: "8b2251dbf94b3f6aca867376d00014ef"
server_key_iv: "de6d3cdbf53170a2008d382a90645d01"
server_key_version: "encryptionkey@0"
I20260925 02:45:11.075026 15501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "f325afc3ad714f93bd2f023035dedd23"
format_stamp: "Formatted at 2026-09-25 02:45:11 on dist-test-slave-n563"
server_key: "8b2251dbf94b3f6aca867376d00014ef"
server_key_iv: "de6d3cdbf53170a2008d382a90645d01"
server_key_version: "encryptionkey@0"
I20260925 02:45:11.080366 15501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:45:11.083997 15515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:11.085348 15501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:45:11.085549 15501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "f325afc3ad714f93bd2f023035dedd23"
format_stamp: "Formatted at 2026-09-25 02:45:11 on dist-test-slave-n563"
server_key: "8b2251dbf94b3f6aca867376d00014ef"
server_key_iv: "de6d3cdbf53170a2008d382a90645d01"
server_key_version: "encryptionkey@0"
I20260925 02:45:11.085798 15501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:11.095290 15501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:11.096356 15501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:11.096674 15501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:11.097813 15501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:45:11.099728 15501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:45:11.099834 15501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:11.099972 15501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:45:11.100044 15501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:11.136358 15501 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:37159
I20260925 02:45:11.136458 15628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:37159 every 8 connection(s)
I20260925 02:45:11.137719 15501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:45:11.146090 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:11.140301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:54073 (local address 127.5.56.190:36631)
0925 02:45:11.140493 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:11.140497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:11.141925 (+ 1428us) server_negotiation.cc:407] Connection header received
0925 02:45:11.143680 (+ 1755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:11.143685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:11.143768 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:11.143886 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:11.145250 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:11.145256 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:11.145261 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:11.145288 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:11.145321 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:11.145332 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:11.145465 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:11.145665 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 02:45:11.145742 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:45:11.147220 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:11.140813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36631 (local address 127.5.56.130:54073)
0925 02:45:11.141757 (+ 944us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:11.141795 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:11.143392 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:11.144138 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:11.144146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:11.144240 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:11.144942 (+ 702us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:45:11.144945 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:11.144983 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:45:11.145070 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:45:11.145441 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:45:11.145452 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:45:11.145459 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:45:11.145749 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 02:45:11.146044 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":301,"threads_started":1}
I20260925 02:45:11.147382 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 15501
I20260925 02:45:11.147570 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:45:11.148015 5346 external_mini_cluster.cc:1428] Setting key a1087bf1d3611540e0ac595cfa2a3ec5
I20260925 02:45:11.148867 15629 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36631
I20260925 02:45:11.149201 15629 heartbeater.cc:464] Registering TS with master...
I20260925 02:45:11.149825 15629 heartbeater.cc:511] Master 127.5.56.190:36631 requested a full tablet report, sending...
I20260925 02:45:11.150790 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36631
--builtin_ntp_servers=127.5.56.148:43477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260925 02:45:11.151367 15309 ts_manager.cc:194] Registered new tserver with Master: f325afc3ad714f93bd2f023035dedd23 (127.5.56.130:37159)
W20260925 02:45:11.499951 15632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:11.500334 15632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:11.500521 15632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:11.508082 15632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:11.508260 15632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:11.508430 15632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:45:11.517946 15632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:11.519433 15632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:11.521363 15632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:11.535753 15640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:11.535823 15637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:11.535838 15638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:11.536918 15632 server_base.cc:1034] running on GCE node
I20260925 02:45:11.537556 15632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:11.538595 15632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:11.539804 15632 hybrid_clock.cc:648] HybridClock initialized: now 1790304311539762 us; error 33 us; skew 500 ppm
I20260925 02:45:11.540166 15632 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:45:11.542678 15632 webserver.cc:460] Webserver started at http://127.5.56.131:42885/ using document root <none> and password file <none>
I20260925 02:45:11.543632 15632 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:11.543761 15632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:11.544211 15632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:11.547916 15632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "c43da0a109f64c108b0566d56471befe"
format_stamp: "Formatted at 2026-09-25 02:45:11 on dist-test-slave-n563"
server_key: "08a8be60b791b39c2d58cb36b375e65c"
server_key_iv: "0b73ee53d2563e575026925178bc3f30"
server_key_version: "encryptionkey@0"
I20260925 02:45:11.548801 15632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "c43da0a109f64c108b0566d56471befe"
format_stamp: "Formatted at 2026-09-25 02:45:11 on dist-test-slave-n563"
server_key: "08a8be60b791b39c2d58cb36b375e65c"
server_key_iv: "0b73ee53d2563e575026925178bc3f30"
server_key_version: "encryptionkey@0"
I20260925 02:45:11.554373 15632 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:45:11.558185 15646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:11.559727 15632 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:45:11.559924 15632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "c43da0a109f64c108b0566d56471befe"
format_stamp: "Formatted at 2026-09-25 02:45:11 on dist-test-slave-n563"
server_key: "08a8be60b791b39c2d58cb36b375e65c"
server_key_iv: "0b73ee53d2563e575026925178bc3f30"
server_key_version: "encryptionkey@0"
I20260925 02:45:11.560169 15632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:11.566460 15632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:11.567337 15632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:11.567631 15632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:11.568594 15632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:45:11.570098 15632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:45:11.570228 15632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:11.570358 15632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:45:11.570487 15632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 02:45:11.605427 15632 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:33857
I20260925 02:45:11.605470 15759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:33857 every 8 connection(s)
I20260925 02:45:11.607146 15632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:45:11.608309 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 15632
I20260925 02:45:11.608527 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:45:11.609092 5346 external_mini_cluster.cc:1428] Setting key 2282944a9dbb99b60772e11c995fcc76
I20260925 02:45:11.616986 15499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:11.610811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:52109 (local address 127.5.56.190:36631)
0925 02:45:11.611034 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:11.611040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:11.612926 (+ 1886us) server_negotiation.cc:407] Connection header received
0925 02:45:11.614774 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:11.614779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:11.614836 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:11.614933 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:11.616314 (+ 1381us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:11.616319 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:11.616322 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:11.616339 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:11.616372 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:11.616384 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:11.616479 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:11.616666 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 02:45:11.616728 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 02:45:11.617998 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:11.611583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36631 (local address 127.5.56.131:52109)
0925 02:45:11.612754 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:11.612792 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:11.614513 (+ 1721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:11.615166 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:11.615174 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:11.615274 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:11.616015 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:45:11.616018 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:11.616059 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:45:11.616155 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:45:11.616451 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:45:11.616456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:45:11.616461 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:45:11.616676 (+ 215us) client_negotiation.cc:240] Negotiation successful
0925 02:45:11.616926 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":391,"threads_started":1}
I20260925 02:45:11.619637 15760 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36631
I20260925 02:45:11.619971 15760 heartbeater.cc:464] Registering TS with master...
I20260925 02:45:11.620605 15760 heartbeater.cc:511] Master 127.5.56.190:36631 requested a full tablet report, sending...
I20260925 02:45:11.622169 15309 ts_manager.cc:194] Registered new tserver with Master: c43da0a109f64c108b0566d56471befe (127.5.56.131:33857)
I20260925 02:45:11.625423 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 02:45:11.632144 15499 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:45:11.630328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.5.56.190:36631)
0925 02:45:11.630594 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:11.630601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:11.631133 (+ 532us) server_negotiation.cc:407] Connection header received
0925 02:45:11.631349 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:11.631355 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:11.631431 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:11.631543 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:11.631911 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43004: BlockingRecv error: recv got EOF from 127.0.0.1:43004 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260925 02:45:11.632890 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 15368
I20260925 02:45:11.636581 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:11.633959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53050 (local address 127.5.56.129:40921)
0925 02:45:11.634534 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:11.634541 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:11.634568 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:45:11.634627 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:11.634634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:11.634860 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:11.635058 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:11.635882 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:11.635889 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:11.635907 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:11.635924 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:11.635964 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:11.635978 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:11.636083 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:11.636165 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:45:11.636278 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":222,"threads_started":1}
I20260925 02:45:11.637868 15492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:11.719715 15497 heartbeater.cc:503] Master 127.5.56.190:36631 was elected leader, sending a full tablet report...
I20260925 02:45:11.721572 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 15368
W20260925 02:45:11.739213 15293 connection.cc:535] server connection from 127.5.56.129:51939 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:45:11.739521 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 15501
I20260925 02:45:11.743348 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:11.740598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49018 (local address 127.5.56.130:37159)
0925 02:45:11.741205 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:11.741213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:11.741246 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:45:11.741309 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:11.741315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:11.741552 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:11.741679 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:11.742724 (+ 1045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:11.742731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:11.742743 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:11.742764 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:11.742811 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:11.742825 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:11.742959 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:11.743013 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 02:45:11.743098 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":328,"threads_started":1}
I20260925 02:45:11.744683 15624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:11.828228 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 15501
I20260925 02:45:11.846051 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 15632
I20260925 02:45:11.850370 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:11.847063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42260 (local address 127.5.56.131:33857)
0925 02:45:11.847664 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:11.847672 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:11.847703 (+ 31us) server_negotiation.cc:407] Connection header received
0925 02:45:11.847766 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:11.847773 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:11.847999 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:11.848183 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:11.849527 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:11.849534 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:11.849546 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:45:11.849559 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:11.849605 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:45:11.849622 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:11.849842 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:11.849914 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:45:11.850069 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":273,"threads_started":1}
I20260925 02:45:11.851786 15755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:11.935730 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 15632
I20260925 02:45:11.953150 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 15276
I20260925 02:45:11.954599 15339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:12.059624 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 15276
2026-09-25T02:45:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2486 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:45:12 dist-test-slave-n563 krb5kdc[15778](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:45:12 dist-test-slave-n563 krb5kdc[15778](info): set up 2 sockets
Sep 25 02:45:12 dist-test-slave-n563 krb5kdc[15778](info): commencing operation
krb5kdc: starting...
W20260925 02:45:13.180719 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:45:13 dist-test-slave-n563 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:45:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:45:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:45:13.432144 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:36161
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:36161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:45:13.770816 15791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:13.771216 15791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:13.771391 15791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:13.778864 15791 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:45:13.779049 15791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:13.779173 15791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:13.779285 15791 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:45:13.779376 15791 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:45:13.789554 15791 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:36161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:36161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:13.791059 15791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:13.792451 15791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:13.804437 15797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:13.804574 15798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:13.804733 15800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:13.805471 15791 server_base.cc:1034] running on GCE node
I20260925 02:45:13.806303 15791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:13.807806 15791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:13.809042 15791 hybrid_clock.cc:648] HybridClock initialized: now 1790304313808996 us; error 58 us; skew 500 ppm
I20260925 02:45:13.809417 15791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:45:13 dist-test-slave-n563 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304313, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:45:13.819221 15791 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:45:13.820897 15791 webserver.cc:460] Webserver started at http://127.5.56.190:44959/ using document root <none> and password file <none>
I20260925 02:45:13.821679 15791 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:13.821795 15791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:13.822077 15791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:13.825528 15791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "e7ada9f5357d43a79a51d30b39282c2c"
format_stamp: "Formatted at 2026-09-25 02:45:13 on dist-test-slave-n563"
server_key: "c9b5e34e09cb68e2e1424fbc60276f43"
server_key_iv: "9b92c3867e97176c62a0a5a43336ccb8"
server_key_version: "encryptionkey@0"
I20260925 02:45:13.826327 15791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "e7ada9f5357d43a79a51d30b39282c2c"
format_stamp: "Formatted at 2026-09-25 02:45:13 on dist-test-slave-n563"
server_key: "c9b5e34e09cb68e2e1424fbc60276f43"
server_key_iv: "9b92c3867e97176c62a0a5a43336ccb8"
server_key_version: "encryptionkey@0"
I20260925 02:45:13.832294 15791 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:45:13.836066 15807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:13.837571 15791 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 02:45:13.837780 15791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "e7ada9f5357d43a79a51d30b39282c2c"
format_stamp: "Formatted at 2026-09-25 02:45:13 on dist-test-slave-n563"
server_key: "c9b5e34e09cb68e2e1424fbc60276f43"
server_key_iv: "9b92c3867e97176c62a0a5a43336ccb8"
server_key_version: "encryptionkey@0"
I20260925 02:45:13.838008 15791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:13.871894 15791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:13.876451 15791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:13.876801 15791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:13.897782 15791 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:36161
I20260925 02:45:13.897814 15858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:36161 every 8 connection(s)
I20260925 02:45:13.899317 15791 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:45:13.903426 15859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:45:13.907176 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 15791
I20260925 02:45:13.907410 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:45:13.908062 5346 external_mini_cluster.cc:1428] Setting key e39fc96423e142c8cb6865964a0d4569
I20260925 02:45:13.916364 15859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c: Bootstrap starting.
I20260925 02:45:13.920241 15859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c: Neither blocks nor log segments found. Creating new log.
I20260925 02:45:13.921650 15859 log.cc:824] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:45:13.925122 15859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c: No bootstrap required, opened a new log
Sep 25 02:45:13 dist-test-slave-n563 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:45:13.934659 15862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:13.910303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60068 (local address 127.5.56.190:36161)
0925 02:45:13.911201 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:13.911217 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:13.911270 (+ 53us) server_negotiation.cc:407] Connection header received
0925 02:45:13.914451 (+ 3181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:13.914485 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:13.915055 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:13.915523 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:13.917145 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:13.919233 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:13.920703 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:13.921819 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:13.922023 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:13.926648 (+ 4625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:13.926657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:13.926674 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:45:13.926714 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:13.929966 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:13.930920 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:13.930929 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:13.930936 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:13.931198 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:13.931615 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:13.931619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:13.931621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:13.932260 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:13.932502 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:13.932984 (+ 482us) server_negotiation.cc:299] Negotiation successful
0925 02:45:13.933432 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":348,"threads_started":1}
I20260925 02:45:13.935070 15859 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ada9f5357d43a79a51d30b39282c2c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36161 } }
I20260925 02:45:13.935487 15859 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:45:13.935621 15859 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7ada9f5357d43a79a51d30b39282c2c, State: Initialized, Role: FOLLOWER
I20260925 02:45:13.936273 15859 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ada9f5357d43a79a51d30b39282c2c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36161 } }
I20260925 02:45:13.936553 15859 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:45:13.936712 15859 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:45:13.936908 15859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:45:13.939498 15859 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ada9f5357d43a79a51d30b39282c2c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36161 } }
I20260925 02:45:13.940016 15859 leader_election.cc:302] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7ada9f5357d43a79a51d30b39282c2c; no voters:
I20260925 02:45:13.940600 15859 leader_election.cc:288] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:45:13.940752 15864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:45:13.941473 15864 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [term 1 LEADER]: Becoming Leader. State: Replica: e7ada9f5357d43a79a51d30b39282c2c, State: Running, Role: LEADER
I20260925 02:45:13.942140 15864 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ada9f5357d43a79a51d30b39282c2c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36161 } }
I20260925 02:45:13.943053 15859 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:45:13.948022 15866 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7ada9f5357d43a79a51d30b39282c2c. Latest consensus state: current_term: 1 leader_uuid: "e7ada9f5357d43a79a51d30b39282c2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ada9f5357d43a79a51d30b39282c2c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36161 } } }
I20260925 02:45:13.948566 15866 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [sys.catalog]: This master's current role is: LEADER
I20260925 02:45:13.948410 15865 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7ada9f5357d43a79a51d30b39282c2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7ada9f5357d43a79a51d30b39282c2c" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36161 } } }
I20260925 02:45:13.948783 15865 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c [sys.catalog]: This master's current role is: LEADER
I20260925 02:45:13.949510 15880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:45:13.955273 15880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:45:13.964790 15880 catalog_manager.cc:1347] Generated new cluster ID: c17a91d28ffc404bbb549db7216f05c6
I20260925 02:45:13.964936 15880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:45:13.981575 15880 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:45:13.983068 15880 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:45:13.992537 15880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c: Generated new TSK 0
I20260925 02:45:13.993525 15880 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:45:14.061834 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36161
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:45:14.440440 15886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:14.440757 15886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:14.440857 15886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:14.448864 15886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:14.449025 15886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:14.449256 15886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:45:14.459461 15886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:14.460798 15886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:14.462131 15886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:14.475903 15894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:14.475991 15892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:14.475934 15891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:14.476883 15886 server_base.cc:1034] running on GCE node
I20260925 02:45:14.477552 15886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:14.478636 15886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:14.479875 15886 hybrid_clock.cc:648] HybridClock initialized: now 1790304314479805 us; error 56 us; skew 500 ppm
I20260925 02:45:14.480224 15886 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:45:14 dist-test-slave-n563 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304314, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:45:14.483541 15886 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:45:14.485163 15886 webserver.cc:460] Webserver started at http://127.5.56.129:40917/ using document root <none> and password file <none>
I20260925 02:45:14.485901 15886 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:14.486054 15886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:14.486372 15886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:14.489660 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "49132cc3e2484ef2b1797d14791c051e"
format_stamp: "Formatted at 2026-09-25 02:45:14 on dist-test-slave-n563"
server_key: "d304e504985636f7bb2b6423eaa6d708"
server_key_iv: "d5cfb4b6787db4d686b12ce8e106d670"
server_key_version: "encryptionkey@0"
I20260925 02:45:14.490598 15886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "49132cc3e2484ef2b1797d14791c051e"
format_stamp: "Formatted at 2026-09-25 02:45:14 on dist-test-slave-n563"
server_key: "d304e504985636f7bb2b6423eaa6d708"
server_key_iv: "d5cfb4b6787db4d686b12ce8e106d670"
server_key_version: "encryptionkey@0"
I20260925 02:45:14.496376 15886 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:45:14.500260 15901 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:14.501680 15886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:45:14.501871 15886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "49132cc3e2484ef2b1797d14791c051e"
format_stamp: "Formatted at 2026-09-25 02:45:14 on dist-test-slave-n563"
server_key: "d304e504985636f7bb2b6423eaa6d708"
server_key_iv: "d5cfb4b6787db4d686b12ce8e106d670"
server_key_version: "encryptionkey@0"
I20260925 02:45:14.502099 15886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:14.516901 15886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:14.521243 15886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:14.521566 15886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:14.522640 15886 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:45:14.524051 15886 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:45:14.524159 15886 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:14.524317 15886 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:45:14.524395 15886 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:14.560400 15886 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:37769
I20260925 02:45:14.560434 16014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:37769 every 8 connection(s)
I20260925 02:45:14.561715 15886 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:45:14.571488 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 15886
I20260925 02:45:14.571718 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:45:14.572535 5346 external_mini_cluster.cc:1428] Setting key f92ecf2eb27c1cdd91014e09c08cfd22
Sep 25 02:45:14 dist-test-slave-n563 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304314, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:45:14.583405 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:14.564270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:38919 (local address 127.5.56.190:36161)
0925 02:45:14.564684 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:14.564689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:14.565709 (+ 1020us) server_negotiation.cc:407] Connection header received
0925 02:45:14.567733 (+ 2024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:14.567739 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:14.567832 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:14.567970 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:14.569647 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:14.570350 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:14.572506 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:14.572967 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:14.573066 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:14.577047 (+ 3981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:14.577052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:14.577055 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:45:14.577087 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:14.579741 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:14.580470 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:14.580476 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:14.580481 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:14.580616 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:14.581053 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:14.581058 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:14.581062 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:14.581450 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:14.581632 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:14.582659 (+ 1027us) server_negotiation.cc:299] Negotiation successful
0925 02:45:14.582914 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":200,"threads_started":1}
I20260925 02:45:14.584126 16018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:14.564829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36161 (local address 127.5.56.129:38919)
0925 02:45:14.565545 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:14.565588 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:14.567396 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:14.568323 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:14.568334 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:14.568893 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:14.569472 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:14.569489 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:14.570524 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:14.570528 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:14.572347 (+ 1819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:14.572365 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:14.573204 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:14.573209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:14.573465 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:14.574225 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:14.574248 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:14.576818 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:45:14.579886 (+ 3068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:45:14.579895 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:45:14.579912 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:45:14.580346 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:45:14.580731 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:45:14.580736 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:45:14.580740 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:45:14.580938 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:45:14.581615 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:45:14.581626 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:45:14.582118 (+ 492us) client_negotiation.cc:769] Sending connection context
0925 02:45:14.582556 (+ 438us) client_negotiation.cc:240] Negotiation successful
0925 02:45:14.582976 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":282,"threads_started":1}
I20260925 02:45:14.585711 16015 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36161
I20260925 02:45:14.586035 16015 heartbeater.cc:464] Registering TS with master...
I20260925 02:45:14.586861 16015 heartbeater.cc:511] Master 127.5.56.190:36161 requested a full tablet report, sending...
I20260925 02:45:14.588985 15824 ts_manager.cc:194] Registered new tserver with Master: 49132cc3e2484ef2b1797d14791c051e (127.5.56.129:37769)
I20260925 02:45:14.590556 15824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:38919
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:45:14.625314 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36161
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:45:14.960546 16021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:14.960829 16021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:14.960924 16021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:14.968262 16021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:14.968376 16021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:14.968499 16021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:45:14.978340 16021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:14.979650 16021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:14.980911 16021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:14.994616 16026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:14.994555 16027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:14.994656 16029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:14.995477 16021 server_base.cc:1034] running on GCE node
I20260925 02:45:14.996043 16021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:14.996935 16021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:14.998142 16021 hybrid_clock.cc:648] HybridClock initialized: now 1790304314998083 us; error 43 us; skew 500 ppm
I20260925 02:45:14.998526 16021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:45:14 dist-test-slave-n563 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304314, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:45:15.001710 16021 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:45:15.003142 16021 webserver.cc:460] Webserver started at http://127.5.56.130:40591/ using document root <none> and password file <none>
I20260925 02:45:15.003793 16021 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:15.003888 16021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:15.004154 16021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:15.007315 16021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "3ca78dbc667146ee9460ea5c40f83f41"
format_stamp: "Formatted at 2026-09-25 02:45:15 on dist-test-slave-n563"
server_key: "5cd23523b99cb4208f1bf13d57980f3c"
server_key_iv: "57e4f95bb1ad69d9cfca9f28233c5408"
server_key_version: "encryptionkey@0"
I20260925 02:45:15.008109 16021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "3ca78dbc667146ee9460ea5c40f83f41"
format_stamp: "Formatted at 2026-09-25 02:45:15 on dist-test-slave-n563"
server_key: "5cd23523b99cb4208f1bf13d57980f3c"
server_key_iv: "57e4f95bb1ad69d9cfca9f28233c5408"
server_key_version: "encryptionkey@0"
I20260925 02:45:15.013403 16021 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:45:15.017001 16036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:15.018456 16021 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:45:15.018653 16021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "3ca78dbc667146ee9460ea5c40f83f41"
format_stamp: "Formatted at 2026-09-25 02:45:15 on dist-test-slave-n563"
server_key: "5cd23523b99cb4208f1bf13d57980f3c"
server_key_iv: "57e4f95bb1ad69d9cfca9f28233c5408"
server_key_version: "encryptionkey@0"
I20260925 02:45:15.018858 16021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:15.037566 16021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:15.042318 16021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:15.042645 16021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:15.043788 16021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:45:15.045204 16021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:45:15.045344 16021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:15.045531 16021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:45:15.045604 16021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:15.079841 16021 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:39599
I20260925 02:45:15.079906 16151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:39599 every 8 connection(s)
I20260925 02:45:15.081162 16021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:45:15.089950 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 16021
I20260925 02:45:15.090168 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:45:15.090746 5346 external_mini_cluster.cc:1428] Setting key 76f81f0993b69e0aa531db177db22516
Sep 25 02:45:15 dist-test-slave-n563 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304314, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:45:15.102289 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.083733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:45103 (local address 127.5.56.190:36161)
0925 02:45:15.083943 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:15.083950 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:15.085261 (+ 1311us) server_negotiation.cc:407] Connection header received
0925 02:45:15.087076 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:15.087081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:15.087152 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:15.087244 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:15.088779 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.089533 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.091698 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.092185 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.092325 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.096022 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:15.096028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:15.096031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:45:15.096065 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:15.099001 (+ 2936us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:15.099839 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:15.099852 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:15.099855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:15.099999 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:15.100471 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:15.100476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:15.100478 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:15.100763 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:15.100949 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:15.101667 (+ 718us) server_negotiation.cc:299] Negotiation successful
0925 02:45:15.101920 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:45:15.103395 16154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.084268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36161 (local address 127.5.56.130:45103)
0925 02:45:15.085095 (+ 827us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:15.085140 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:15.086821 (+ 1681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:15.087455 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:15.087463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:15.087968 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:15.088538 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.088556 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.089688 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.089696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.091572 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.091584 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.092424 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.092431 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.092737 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.093517 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:15.093543 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:15.095777 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:45:15.099176 (+ 3399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:45:15.099186 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:45:15.099205 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:45:15.099685 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:45:15.100122 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:45:15.100127 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:45:15.100131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:45:15.100339 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:45:15.100897 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:45:15.100906 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:45:15.101297 (+ 391us) client_negotiation.cc:769] Sending connection context
0925 02:45:15.101575 (+ 278us) client_negotiation.cc:240] Negotiation successful
0925 02:45:15.102170 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":336,"threads_started":1}
I20260925 02:45:15.105001 16152 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36161
I20260925 02:45:15.105329 16152 heartbeater.cc:464] Registering TS with master...
I20260925 02:45:15.106240 16152 heartbeater.cc:511] Master 127.5.56.190:36161 requested a full tablet report, sending...
I20260925 02:45:15.107798 15824 ts_manager.cc:194] Registered new tserver with Master: 3ca78dbc667146ee9460ea5c40f83f41 (127.5.56.130:39599)
I20260925 02:45:15.109019 15824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:45103
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:45:15.145756 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36161
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:44775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:45:15.484408 16157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:45:15.484699 16157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:45:15.484793 16157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:45:15.492187 16157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:45:15.492307 16157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:45:15.492420 16157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:45:15.502416 16157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:44775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36161
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:45:15.503734 16157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:45:15.505105 16157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:45:15.518882 16163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:15.518980 16165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:45:15.518960 16162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:45:15.519716 16157 server_base.cc:1034] running on GCE node
I20260925 02:45:15.520342 16157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:45:15.521319 16157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:45:15.522562 16157 hybrid_clock.cc:648] HybridClock initialized: now 1790304315522506 us; error 41 us; skew 500 ppm
I20260925 02:45:15.522943 16157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:45:15 dist-test-slave-n563 krb5kdc[15778](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304315, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:45:15.526217 16157 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:45:15.527798 16157 webserver.cc:460] Webserver started at http://127.5.56.131:34703/ using document root <none> and password file <none>
I20260925 02:45:15.528541 16157 fs_manager.cc:359] Metadata directory not provided
I20260925 02:45:15.528692 16157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:45:15.529016 16157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:45:15.532060 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "74ad4dd20ee7460ea01f69313b898eb0"
format_stamp: "Formatted at 2026-09-25 02:45:15 on dist-test-slave-n563"
server_key: "faaf03ecb22eb61ca605019acac8de00"
server_key_iv: "a57ef4a707c167009744b966ddb50c6b"
server_key_version: "encryptionkey@0"
I20260925 02:45:15.532873 16157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "74ad4dd20ee7460ea01f69313b898eb0"
format_stamp: "Formatted at 2026-09-25 02:45:15 on dist-test-slave-n563"
server_key: "faaf03ecb22eb61ca605019acac8de00"
server_key_iv: "a57ef4a707c167009744b966ddb50c6b"
server_key_version: "encryptionkey@0"
I20260925 02:45:15.538214 16157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:45:15.541689 16172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:15.543243 16157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:45:15.543416 16157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "74ad4dd20ee7460ea01f69313b898eb0"
format_stamp: "Formatted at 2026-09-25 02:45:15 on dist-test-slave-n563"
server_key: "faaf03ecb22eb61ca605019acac8de00"
server_key_iv: "a57ef4a707c167009744b966ddb50c6b"
server_key_version: "encryptionkey@0"
I20260925 02:45:15.543630 16157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:45:15.554508 16157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:45:15.559124 16157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:45:15.559458 16157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:45:15.560896 16157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:45:15.562739 16157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:45:15.562815 16157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:15.562906 16157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:45:15.562948 16157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:45:15.593531 16015 heartbeater.cc:503] Master 127.5.56.190:36161 was elected leader, sending a full tablet report...
I20260925 02:45:15.597745 16157 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:45947
I20260925 02:45:15.597786 16285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:45947 every 8 connection(s)
I20260925 02:45:15.599121 16157 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:45:15.599931 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 16157
I20260925 02:45:15.600135 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:45:15.600667 5346 external_mini_cluster.cc:1428] Setting key d08529c698049c368c2f2bb0e0e2f42a
Sep 25 02:45:15 dist-test-slave-n563 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304315, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:45:15.619876 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.602015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:59197 (local address 127.5.56.190:36161)
0925 02:45:15.602222 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:15.602228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:15.603686 (+ 1458us) server_negotiation.cc:407] Connection header received
0925 02:45:15.605842 (+ 2156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:15.605848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:15.605976 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:15.606086 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:15.607674 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.608421 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.609976 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.610435 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.610571 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.613955 (+ 3384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:15.613960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:15.613964 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:45:15.613998 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:15.616753 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:15.617497 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:15.617504 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:15.617508 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:15.617648 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:15.618076 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:15.618085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:15.618088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:15.618369 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:15.618583 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:15.619279 (+ 696us) server_negotiation.cc:299] Negotiation successful
0925 02:45:15.619501 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:45:15.620883 16288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.602672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36161 (local address 127.5.56.131:59197)
0925 02:45:15.603546 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:15.603583 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:15.605571 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:15.606334 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:15.606342 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:15.606938 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:45:15.607510 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.607527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.608577 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.608583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.609824 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.609840 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.610675 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.610682 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.610936 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.611538 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:45:15.611560 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:45:15.613730 (+ 2170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:45:15.616904 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:45:15.616913 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:45:15.616931 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:45:15.617369 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:45:15.617763 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:45:15.617768 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:45:15.617772 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:45:15.617964 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:45:15.618552 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:45:15.618560 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:45:15.618928 (+ 368us) client_negotiation.cc:769] Sending connection context
0925 02:45:15.619203 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 02:45:15.619772 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":399,"threads_started":1}
I20260925 02:45:15.622316 16286 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36161
I20260925 02:45:15.622725 16286 heartbeater.cc:464] Registering TS with master...
I20260925 02:45:15.623468 16286 heartbeater.cc:511] Master 127.5.56.190:36161 requested a full tablet report, sending...
I20260925 02:45:15.624989 15824 ts_manager.cc:194] Registered new tserver with Master: 74ad4dd20ee7460ea01f69313b898eb0 (127.5.56.131:45947)
I20260925 02:45:15.625795 15824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:59197
I20260925 02:45:15.628940 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:45:15.646298 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.633465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60074 (local address 127.5.56.190:36161)
0925 02:45:15.633691 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:15.633701 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:15.634164 (+ 463us) server_negotiation.cc:407] Connection header received
0925 02:45:15.634316 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:15.634324 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:15.634471 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:15.634589 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:15.636014 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.636832 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.638058 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.638496 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.638594 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.640748 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:15.640752 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:15.640756 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:45:15.640792 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:15.643621 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:15.644264 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:15.644277 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:15.644279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:15.644404 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:15.644809 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:15.644815 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:15.644819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:15.645139 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:15.645359 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:15.645780 (+ 421us) server_negotiation.cc:299] Negotiation successful
0925 02:45:15.645930 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:45:15.650184 15824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:45:15.657055 15824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:45:15.692823 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.669595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59324 (local address 127.5.56.131:45947)
0925 02:45:15.681007 (+ 11412us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:15.681021 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:15.681067 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:45:15.681198 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:15.681211 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:15.681535 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:15.681768 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:45:15.684931 (+ 3163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.686610 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.689683 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.691458 (+ 1775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.691610 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.691892 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:15.692268 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 02:45:15.692445 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11169,"thread_start_us":10956,"threads_started":1}
I20260925 02:45:15.692866 16300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.668798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45947 (local address 127.0.0.1:59324)
0925 02:45:15.679017 (+ 10219us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:15.679059 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:15.679262 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:15.682367 (+ 3105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:15.682403 (+ 36us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:15.682447 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:45:15.682967 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.682994 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.686808 (+ 3814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.686814 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.689537 (+ 2723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.689554 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.691679 (+ 2125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.691687 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.692014 (+ 327us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.692060 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:45:15.692219 (+ 159us) client_negotiation.cc:240] Negotiation successful
0925 02:45:15.692441 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9933,"mutex_wait_us":59,"thread_start_us":17620,"threads_started":1}
I20260925 02:45:15.695225 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.668900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39599 (local address 127.0.0.1:50566)
0925 02:45:15.684930 (+ 16030us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:15.684952 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:15.685069 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:15.687218 (+ 2149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:15.687224 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:15.687257 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:45:15.687699 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.687717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.689675 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.689681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.692401 (+ 2720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.692421 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.694360 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.694366 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.694544 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.694575 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:45:15.694673 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 02:45:15.694804 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15906,"mutex_wait_us":118,"thread_start_us":15758,"threads_started":1}
I20260925 02:45:15.695240 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.670172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50566 (local address 127.5.56.130:39599)
0925 02:45:15.686286 (+ 16114us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:15.686294 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:15.686334 (+ 40us) server_negotiation.cc:407] Connection header received
0925 02:45:15.686458 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:15.686465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:15.686723 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:15.686944 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:45:15.687925 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.689465 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.692616 (+ 3151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.694136 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.694286 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.694549 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:15.694733 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 02:45:15.694909 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15966,"thread_start_us":15818,"threads_started":1}
I20260925 02:45:15.696396 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.669459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:37769 (local address 127.0.0.1:35428)
0925 02:45:15.686697 (+ 17238us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:15.686717 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:15.686852 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:15.687794 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:15.687799 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:15.687823 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:45:15.688211 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.688219 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.689594 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.689602 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.692440 (+ 2838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.692450 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.695836 (+ 3386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.695841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.695955 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.695975 (+ 20us) client_negotiation.cc:769] Sending connection context
0925 02:45:15.696046 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 02:45:15.696135 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17090,"mutex_wait_us":182,"thread_start_us":9173,"threads_started":1}
I20260925 02:45:15.696779 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.670148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35428 (local address 127.5.56.129:37769)
0925 02:45:15.685649 (+ 15501us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:15.685659 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:15.687018 (+ 1359us) server_negotiation.cc:407] Connection header received
0925 02:45:15.687138 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:15.687148 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:15.687371 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:15.687553 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:45:15.688407 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.689422 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.692616 (+ 3194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.695407 (+ 2791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.695549 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.695732 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:15.696137 (+ 405us) server_negotiation.cc:299] Negotiation successful
0925 02:45:15.696300 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15303,"thread_start_us":15518,"threads_started":1}
I20260925 02:45:15.697259 16220 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea26dc6c55645b78de694bec52f2bb2 (DEFAULT_TABLE table=test-table [id=4889abe4f6fa422dac352dcddde601a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:45:15.698843 16220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea26dc6c55645b78de694bec52f2bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:45:15.700455 15949 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea26dc6c55645b78de694bec52f2bb2 (DEFAULT_TABLE table=test-table [id=4889abe4f6fa422dac352dcddde601a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:45:15.701649 15949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea26dc6c55645b78de694bec52f2bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:45:15.701932 16086 tablet_service.cc:1467] Processing CreateTablet for tablet 1ea26dc6c55645b78de694bec52f2bb2 (DEFAULT_TABLE table=test-table [id=4889abe4f6fa422dac352dcddde601a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:45:15.703336 16086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ea26dc6c55645b78de694bec52f2bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:45:15.714251 16305 tablet_bootstrap.cc:492] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: Bootstrap starting.
I20260925 02:45:15.718212 16304 tablet_bootstrap.cc:492] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: Bootstrap starting.
I20260925 02:45:15.717522 16306 tablet_bootstrap.cc:492] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: Bootstrap starting.
I20260925 02:45:15.721177 16304 tablet_bootstrap.cc:654] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: Neither blocks nor log segments found. Creating new log.
I20260925 02:45:15.721288 16306 tablet_bootstrap.cc:654] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: Neither blocks nor log segments found. Creating new log.
I20260925 02:45:15.722460 16306 log.cc:824] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: Log is configured to *not* fsync() on all Append() calls
I20260925 02:45:15.722481 16304 log.cc:824] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:45:15.723021 16305 tablet_bootstrap.cc:654] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: Neither blocks nor log segments found. Creating new log.
I20260925 02:45:15.724341 16305 log.cc:824] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:45:15.725835 16306 tablet_bootstrap.cc:492] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: No bootstrap required, opened a new log
I20260925 02:45:15.725835 16304 tablet_bootstrap.cc:492] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: No bootstrap required, opened a new log
I20260925 02:45:15.726115 16306 ts_tablet_manager.cc:1397] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260925 02:45:15.726115 16304 ts_tablet_manager.cc:1397] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.003s
I20260925 02:45:15.727180 16305 tablet_bootstrap.cc:492] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: No bootstrap required, opened a new log
I20260925 02:45:15.727490 16305 ts_tablet_manager.cc:1397] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260925 02:45:15.735046 16305 raft_consensus.cc:348] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ad4dd20ee7460ea01f69313b898eb0" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45947 } } peers { permanent_uuid: "3ca78dbc667146ee9460ea5c40f83f41" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39599 } } peers { permanent_uuid: "49132cc3e2484ef2b1797d14791c051e" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37769 } }
I20260925 02:45:15.735466 16305 raft_consensus.cc:374] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:45:15.735605 16305 raft_consensus.cc:729] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ad4dd20ee7460ea01f69313b898eb0, State: Initialized, Role: FOLLOWER
I20260925 02:45:15.736248 16305 consensus_queue.cc:260] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ad4dd20ee7460ea01f69313b898eb0" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45947 } } peers { permanent_uuid: "3ca78dbc667146ee9460ea5c40f83f41" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39599 } } peers { permanent_uuid: "49132cc3e2484ef2b1797d14791c051e" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37769 } }
I20260925 02:45:15.737417 16286 heartbeater.cc:503] Master 127.5.56.190:36161 was elected leader, sending a full tablet report...
I20260925 02:45:15.737233 16306 raft_consensus.cc:348] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ad4dd20ee7460ea01f69313b898eb0" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45947 } } peers { permanent_uuid: "3ca78dbc667146ee9460ea5c40f83f41" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39599 } } peers { permanent_uuid: "49132cc3e2484ef2b1797d14791c051e" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37769 } }
I20260925 02:45:15.737233 16304 raft_consensus.cc:348] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ad4dd20ee7460ea01f69313b898eb0" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45947 } } peers { permanent_uuid: "3ca78dbc667146ee9460ea5c40f83f41" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39599 } } peers { permanent_uuid: "49132cc3e2484ef2b1797d14791c051e" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37769 } }
I20260925 02:45:15.737668 16304 raft_consensus.cc:374] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:45:15.737672 16306 raft_consensus.cc:374] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:45:15.737763 16304 raft_consensus.cc:729] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49132cc3e2484ef2b1797d14791c051e, State: Initialized, Role: FOLLOWER
I20260925 02:45:15.737763 16306 raft_consensus.cc:729] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ca78dbc667146ee9460ea5c40f83f41, State: Initialized, Role: FOLLOWER
I20260925 02:45:15.738168 16305 ts_tablet_manager.cc:1428] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
I20260925 02:45:15.738370 16304 consensus_queue.cc:260] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ad4dd20ee7460ea01f69313b898eb0" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45947 } } peers { permanent_uuid: "3ca78dbc667146ee9460ea5c40f83f41" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39599 } } peers { permanent_uuid: "49132cc3e2484ef2b1797d14791c051e" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37769 } }
I20260925 02:45:15.738480 16306 consensus_queue.cc:260] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ad4dd20ee7460ea01f69313b898eb0" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45947 } } peers { permanent_uuid: "3ca78dbc667146ee9460ea5c40f83f41" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39599 } } peers { permanent_uuid: "49132cc3e2484ef2b1797d14791c051e" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37769 } }
I20260925 02:45:15.740281 16306 ts_tablet_manager.cc:1428] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260925 02:45:15.740655 16304 ts_tablet_manager.cc:1428] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: Time spent starting tablet: real 0.014s user 0.008s sys 0.008s
I20260925 02:45:15.740803 16152 heartbeater.cc:503] Master 127.5.56.190:36161 was elected leader, sending a full tablet report...
W20260925 02:45:15.814186 16016 tablet.cc:2369] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:45:15.833381 16153 tablet.cc:2369] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:45:15.850821 16287 tablet.cc:2369] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:45:15.873250 16312 raft_consensus.cc:482] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:45:15.873588 16312 raft_consensus.cc:504] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ad4dd20ee7460ea01f69313b898eb0" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45947 } } peers { permanent_uuid: "3ca78dbc667146ee9460ea5c40f83f41" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39599 } } peers { permanent_uuid: "49132cc3e2484ef2b1797d14791c051e" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37769 } }
I20260925 02:45:15.875465 16312 leader_election.cc:288] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74ad4dd20ee7460ea01f69313b898eb0 (127.5.56.131:45947), 49132cc3e2484ef2b1797d14791c051e (127.5.56.129:37769)
I20260925 02:45:15.884260 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.875778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:50045 (local address 127.5.56.131:45947)
0925 02:45:15.876003 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:15.876013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:15.876583 (+ 570us) server_negotiation.cc:407] Connection header received
0925 02:45:15.876913 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:15.876919 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:15.877000 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:15.877098 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:45:15.877906 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.879051 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.882169 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.883167 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.883324 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.883395 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:15.883841 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 02:45:15.883991 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:45:15.884354 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.876314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:42291 (local address 127.5.56.129:37769)
0925 02:45:15.876547 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:15.876552 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:15.876807 (+ 255us) server_negotiation.cc:407] Connection header received
0925 02:45:15.876927 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:15.876931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:15.876999 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:15.877097 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:45:15.877896 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.879044 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.882162 (+ 3118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.883399 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.883579 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.883659 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:15.883953 (+ 294us) server_negotiation.cc:299] Negotiation successful
0925 02:45:15.884088 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260925 02:45:15.884492 16313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.875709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45947 (local address 127.5.56.130:50045)
0925 02:45:15.876458 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:15.876480 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:15.876708 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:15.877135 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:15.877140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:15.877166 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:45:15.877727 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.877747 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.879182 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.879187 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.881925 (+ 2738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.881939 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.883396 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.883403 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.883631 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.883675 (+ 44us) client_negotiation.cc:769] Sending connection context
0925 02:45:15.883798 (+ 123us) client_negotiation.cc:240] Negotiation successful
0925 02:45:15.883963 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":178,"thread_start_us":365,"threads_started":1}
I20260925 02:45:15.884521 16314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.875760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:37769 (local address 127.5.56.130:42291)
0925 02:45:15.876691 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0925 02:45:15.876706 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:45:15.876808 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:45:15.877135 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:45:15.877140 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:45:15.877223 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:45:15.877755 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.877763 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.879183 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.879187 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.881924 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:45:15.881939 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.883611 (+ 1672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.883615 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:45:15.883746 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.883802 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 02:45:15.883895 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 02:45:15.884037 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":313,"spinlock_wait_cycles":22272,"thread_start_us":670,"threads_started":1}
I20260925 02:45:15.885710 16241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea26dc6c55645b78de694bec52f2bb2" candidate_uuid: "3ca78dbc667146ee9460ea5c40f83f41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74ad4dd20ee7460ea01f69313b898eb0" is_pre_election: true
I20260925 02:45:15.885710 15969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea26dc6c55645b78de694bec52f2bb2" candidate_uuid: "3ca78dbc667146ee9460ea5c40f83f41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49132cc3e2484ef2b1797d14791c051e" is_pre_election: true
I20260925 02:45:15.886271 15969 raft_consensus.cc:2454] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ca78dbc667146ee9460ea5c40f83f41 in term 0.
I20260925 02:45:15.886271 16241 raft_consensus.cc:2454] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3ca78dbc667146ee9460ea5c40f83f41 in term 0.
I20260925 02:45:15.887055 16038 leader_election.cc:302] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3ca78dbc667146ee9460ea5c40f83f41, 49132cc3e2484ef2b1797d14791c051e, 74ad4dd20ee7460ea01f69313b898eb0; no voters:
I20260925 02:45:15.887487 16312 raft_consensus.cc:2781] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:45:15.887646 16312 raft_consensus.cc:482] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:45:15.887751 16312 raft_consensus.cc:3037] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:45:15.889991 16312 raft_consensus.cc:504] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ad4dd20ee7460ea01f69313b898eb0" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45947 } } peers { permanent_uuid: "3ca78dbc667146ee9460ea5c40f83f41" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39599 } } peers { permanent_uuid: "49132cc3e2484ef2b1797d14791c051e" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37769 } }
I20260925 02:45:15.890647 16312 leader_election.cc:288] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [CANDIDATE]: Term 1 election: Requested vote from peers 74ad4dd20ee7460ea01f69313b898eb0 (127.5.56.131:45947), 49132cc3e2484ef2b1797d14791c051e (127.5.56.129:37769)
I20260925 02:45:15.891153 16241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea26dc6c55645b78de694bec52f2bb2" candidate_uuid: "3ca78dbc667146ee9460ea5c40f83f41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74ad4dd20ee7460ea01f69313b898eb0"
I20260925 02:45:15.891199 15969 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ea26dc6c55645b78de694bec52f2bb2" candidate_uuid: "3ca78dbc667146ee9460ea5c40f83f41" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49132cc3e2484ef2b1797d14791c051e"
I20260925 02:45:15.891360 16241 raft_consensus.cc:3037] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:45:15.891383 15969 raft_consensus.cc:3037] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:45:15.893503 16241 raft_consensus.cc:2454] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ca78dbc667146ee9460ea5c40f83f41 in term 1.
I20260925 02:45:15.893565 15969 raft_consensus.cc:2454] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3ca78dbc667146ee9460ea5c40f83f41 in term 1.
I20260925 02:45:15.893931 16038 leader_election.cc:302] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3ca78dbc667146ee9460ea5c40f83f41, 74ad4dd20ee7460ea01f69313b898eb0; no voters:
I20260925 02:45:15.894232 16312 raft_consensus.cc:2781] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:45:15.894719 16312 raft_consensus.cc:686] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 1 LEADER]: Becoming Leader. State: Replica: 3ca78dbc667146ee9460ea5c40f83f41, State: Running, Role: LEADER
I20260925 02:45:15.895217 16312 consensus_queue.cc:237] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ad4dd20ee7460ea01f69313b898eb0" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45947 } } peers { permanent_uuid: "3ca78dbc667146ee9460ea5c40f83f41" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39599 } } peers { permanent_uuid: "49132cc3e2484ef2b1797d14791c051e" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37769 } }
I20260925 02:45:15.900225 15822 catalog_manager.cc:5495] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ca78dbc667146ee9460ea5c40f83f41 (127.5.56.130). New cstate: current_term: 1 leader_uuid: "3ca78dbc667146ee9460ea5c40f83f41" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74ad4dd20ee7460ea01f69313b898eb0" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ca78dbc667146ee9460ea5c40f83f41" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49132cc3e2484ef2b1797d14791c051e" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37769 } health_report { overall_health: UNKNOWN } } }
I20260925 02:45:15.941679 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.936558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50572 (local address 127.5.56.130:39599)
0925 02:45:15.936752 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:15.936757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:15.936775 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:45:15.936987 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:15.936991 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:15.937071 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:15.937193 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:45:15.937533 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.938247 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.939528 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:15.939914 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:15.940019 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:15.940529 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:45:15.941050 (+ 521us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:45:15.941158 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:15.941289 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 02:45:15.941387 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:45:15.950179 16241 raft_consensus.cc:1261] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0 [term 1 FOLLOWER]: Refusing update from remote peer 3ca78dbc667146ee9460ea5c40f83f41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:45:15.950232 15969 raft_consensus.cc:1261] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e [term 1 FOLLOWER]: Refusing update from remote peer 3ca78dbc667146ee9460ea5c40f83f41: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:45:15.951272 16312 consensus_queue.cc:1035] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74ad4dd20ee7460ea01f69313b898eb0" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:45:15.951612 16315 consensus_queue.cc:1035] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49132cc3e2484ef2b1797d14791c051e" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:45:15.982582 15823 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:45:15.982872 15823 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60074:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:45:15.986851 15823 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7ada9f5357d43a79a51d30b39282c2c: Sending DeleteTablet for 3 replicas of tablet 1ea26dc6c55645b78de694bec52f2bb2
I20260925 02:45:15.988150 16086 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea26dc6c55645b78de694bec52f2bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:45:15 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:50566
I20260925 02:45:15.988220 16220 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea26dc6c55645b78de694bec52f2bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:45:15 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:59324
I20260925 02:45:15.988441 15949 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ea26dc6c55645b78de694bec52f2bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:45:15 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:35428
I20260925 02:45:15.990861 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 15886
I20260925 02:45:15.998942 16330 tablet_replica.cc:330] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: stopping tablet replica
I20260925 02:45:15.999567 16330 raft_consensus.cc:2229] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:45:16.000232 16330 raft_consensus.cc:2258] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:45:16.002139 16329 tablet_replica.cc:330] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: stopping tablet replica
I20260925 02:45:16.002767 16329 raft_consensus.cc:2229] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:45:16.003334 16329 raft_consensus.cc:2258] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:45:16.007650 16329 ts_tablet_manager.cc:1905] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:45:16.007946 16330 ts_tablet_manager.cc:1905] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:45:16.015667 16330 ts_tablet_manager.cc:1918] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:45:16.016201 16328 tablet_replica.cc:330] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: stopping tablet replica
I20260925 02:45:16.016645 16328 raft_consensus.cc:2229] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:45:16.017299 16328 raft_consensus.cc:2258] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 25 02:45:16 dist-test-slave-n563 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:45:16.021669 16328 ts_tablet_manager.cc:1905] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:45:16.023140 16329 ts_tablet_manager.cc:1918] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:45:16.023305 16329 log.cc:1196] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/1ea26dc6c55645b78de694bec52f2bb2
I20260925 02:45:16.023648 16330 log.cc:1196] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/1ea26dc6c55645b78de694bec52f2bb2
I20260925 02:45:16.024276 16330 ts_tablet_manager.cc:1939] T 1ea26dc6c55645b78de694bec52f2bb2 P 49132cc3e2484ef2b1797d14791c051e: Deleting consensus metadata
I20260925 02:45:16.026356 15811 catalog_manager.cc:4841] TS 49132cc3e2484ef2b1797d14791c051e (127.5.56.129:37769): tablet 1ea26dc6c55645b78de694bec52f2bb2 (table test-table [id=4889abe4f6fa422dac352dcddde601a3]) successfully deleted
I20260925 02:45:16.027114 16329 ts_tablet_manager.cc:1939] T 1ea26dc6c55645b78de694bec52f2bb2 P 74ad4dd20ee7460ea01f69313b898eb0: Deleting consensus metadata
I20260925 02:45:16.028981 15809 catalog_manager.cc:4841] TS 74ad4dd20ee7460ea01f69313b898eb0 (127.5.56.131:45947): tablet 1ea26dc6c55645b78de694bec52f2bb2 (table test-table [id=4889abe4f6fa422dac352dcddde601a3]) successfully deleted
I20260925 02:45:16.030066 16328 ts_tablet_manager.cc:1918] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:45:16.030249 16328 log.cc:1196] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/1ea26dc6c55645b78de694bec52f2bb2
I20260925 02:45:16.030869 16302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:15.992663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.5.56.129:37769)
0925 02:45:15.992889 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:15.992898 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:16.005899 (+ 13001us) server_negotiation.cc:407] Connection header received
0925 02:45:16.006259 (+ 360us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:16.006268 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:16.006487 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:16.006585 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:16.009047 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:16.010903 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:16.013530 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:16.014512 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:16.014641 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:16.021507 (+ 6866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:16.021526 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:16.021546 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:45:16.021590 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:16.025581 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:16.026694 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:16.026707 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:16.026712 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:16.026888 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:16.028573 (+ 1685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:16.028578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:16.028582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:16.029108 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:16.029392 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:16.030039 (+ 647us) server_negotiation.cc:299] Negotiation successful
0925 02:45:16.030446 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:45:16.030917 16328 ts_tablet_manager.cc:1939] T 1ea26dc6c55645b78de694bec52f2bb2 P 3ca78dbc667146ee9460ea5c40f83f41: Deleting consensus metadata
I20260925 02:45:16.031647 16009 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:16.033787 15810 catalog_manager.cc:4841] TS 3ca78dbc667146ee9460ea5c40f83f41 (127.5.56.130:39599): tablet 1ea26dc6c55645b78de694bec52f2bb2 (table test-table [id=4889abe4f6fa422dac352dcddde601a3]) successfully deleted
I20260925 02:45:16.179751 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 15886
I20260925 02:45:16.198190 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 16021
Sep 25 02:45:16 dist-test-slave-n563 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:45:16.214232 16303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:16.199100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.5.56.130:39599)
0925 02:45:16.199307 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:16.199314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:16.199342 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:45:16.199415 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:16.199421 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:16.199519 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:16.199631 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:16.200803 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:16.202106 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:16.203414 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:16.203855 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:16.204028 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:16.208184 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:16.208199 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:16.208214 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:45:16.208263 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:16.211353 (+ 3090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:16.212079 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:16.212086 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:16.212088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:16.212237 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:16.212618 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:16.212623 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:16.212633 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:16.213004 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:16.213368 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:16.213615 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 02:45:16.213833 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:45:16.215005 16147 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:16.329412 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 16021
I20260925 02:45:16.349085 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 16157
Sep 25 02:45:16 dist-test-slave-n563 krb5kdc[15778](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:45:16.363883 16301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:45:16.349933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59340 (local address 127.5.56.131:45947)
0925 02:45:16.350112 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:45:16.350117 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:45:16.350137 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:45:16.350201 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:45:16.350205 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:45:16.350289 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:45:16.350442 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:45:16.351697 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:16.352526 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:16.353716 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:45:16.354122 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:45:16.354231 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:45:16.358623 (+ 4392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:45:16.358640 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:45:16.358657 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:45:16.358695 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:45:16.361441 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:16.362057 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:16.362064 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:16.362066 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:16.362213 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:45:16.362664 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:45:16.362668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:45:16.362671 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:45:16.363030 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:45:16.363263 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 02:45:16.363458 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 02:45:16.363618 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:45:16.364533 16281 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:16.458595 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 16157
I20260925 02:45:16.476640 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 15791
I20260925 02:45:16.477766 15854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:45:16.590272 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 15791
2026-09-25T02:45:16Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4531 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:45:16 dist-test-slave-n563 krb5kdc[16338](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:45:16 dist-test-slave-n563 krb5kdc[16338](info): set up 2 sockets
Sep 25 02:45:16 dist-test-slave-n563 krb5kdc[16338](info): commencing operation
krb5kdc: starting...
W20260925 02:45:17.692422 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:45:17 dist-test-slave-n563 krb5kdc[16338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:45:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:45:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.56.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.56.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/rangeradmin_127.5.56.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/rangeradmin_127.5.56.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.56.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.56.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/rangerlookup_127.5.56.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/rangerlookup_127.5.56.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.56.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.56.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/HTTP_127.5.56.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/HTTP_127.5.56.148@KRBTEST.COM.keytab.
I20260925 02:45:18.029826 5346 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task87C_4R/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/postgres -l logfile start
2026-09-25 02:45:20.558 UTC [16369] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:45:20.558 UTC [16369] LOG: listening on IPv4 address "127.5.56.148", port 33041
2026-09-25 02:45:20.562 UTC [16369] LOG: listening on Unix socket "/tmp/.s.PGSQL.33041"
2026-09-25 02:45:20.585 UTC [16372] LOG: database system was shut down at 2026-09-25 02:45:19 UTC
2026-09-25 02:45:20.588 UTC [16369] LOG: database system is ready to accept connections
I20260925 02:45:21.567981 5346 mini_postgres.cc:96] Postgres bound to 33041
2026-09-25 02:45:21.587 UTC [16380] FATAL: database "slave" does not exist
127.5.56.148:33041 - accepting connections
I20260925 02:45:21.588439 5346 mini_ranger.cc:161] Starting Ranger...
I20260925 02:45:21.617167 5346 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:45:22.108508 5346 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:45:22.108656 5346 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-admin
2026-09-25 02:45:22,402 [I] DB FLAVOR :POSTGRES
2026-09-25 02:45:22,403 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:45:22,403 [I] Checking connection..
2026-09-25 02:45:22,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:45:22,642 [I] Checking connection passed.
2026-09-25 02:45:22,643 [I] --------- Verifying version history table ---------
2026-09-25 02:45:22,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:45:22,903 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:45:22,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:45:23,151 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:45:23,151 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:45:23,152 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:45:23.398 UTC [16445] WARNING: there is no transaction in progress
2026-09-25 02:45:23,404 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:45:23,405 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:45:23,674 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:45:23,674 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:45:23,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:45:23,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:45:24,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n563.c.gcp-upstream.internal','N') ;"
2026-09-25 02:45:24,399 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:45:24,691 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:45:24,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:45:24,943 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:45:24,944 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:45:25,181 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:45:25,434 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:45:25,434 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:45:26.395 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.414 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.430 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.446 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.463 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.482 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.549 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.556 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.572 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.583 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.594 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.608 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.619 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.626 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.633 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:26.640 UTC [16556] WARNING: there is no transaction in progress
2026-09-25 02:45:27,314 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:45:27,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:45:27,561 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:45:27,561 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:45:27,801 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:45:27,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:45:28,062 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:45:28,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:45:28,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:45:28,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n563.c.gcp-upstream.internal';"
2026-09-25 02:45:28,807 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:45:28,807 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:45:29,066 [I] DB_PATCHES have already been applied
I20260925 02:45:29.072145 5346 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-admin:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task87C_4R/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:45:29.072247 5346 mini_ranger.cc:193] Using host: 127.5.56.148
I20260925 02:45:29.088255 5346 mini_ranger.cc:227] Ranger admin URL: http://127.5.56.148:47161
Sep 25, 2026 2:45:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:45:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:45:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:45:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:45:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:45:30 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:45:30 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:45:30.267488 5346 mini_ranger.cc:160] Time spent starting Ranger: real 8.679s user 0.000s sys 0.045s
Sep 25, 2026 2:45:50 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:45:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 2:45:50 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:45:50 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:45:51 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:45:51.936411 5346 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.56.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.56.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/rangerkms_127.5.56.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/rangerkms_127.5.56.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.56.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.56.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.56.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/HTTP_127.5.56.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.56.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/HTTP_127.5.56.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:45:52 dist-test-slave-n563 krb5kdc[16338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304352, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:45:52.138865 5346 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:45:52.165572 5346 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:45:53.113893 5346 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:45:53.114056 5346 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-kms
I20260925 02:45:53.114197 5346 mini_ranger_kms.cc:227] Using postgres at 127.5.56.148:33041
2026-09-25 02:45:54,027 [I] DB FLAVOR :POSTGRES
2026-09-25 02:45:54,028 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:45:54,028 [I] Checking connection
2026-09-25 02:45:54,028 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:45:54,268 [I] connection success
2026-09-25 02:45:54,268 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:45:54,268 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:45:54,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:45:54,529 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:45:54,530 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:45:54,530 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:45:54,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:33041/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:45:54,823 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:45:55.383399 5346 mini_ranger_kms.cc:310] Created kms service
I20260925 02:45:55.630990 5346 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:45:55.681545 5346 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:45:55 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304352, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:45:56.113502 5346 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:45:56.115025 5346 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-kms:/tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:45:56.115126 5346 mini_ranger_kms.cc:240] Using host: 127.5.56.148
I20260925 02:45:56.139463 5346 mini_ranger_kms.cc:276] Ranger KMS PID: 16740
I20260925 02:45:56.139671 5346 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.56.148:42015
Sep 25, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:45:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 25 02:45:58 dist-test-slave-n563 krb5kdc[16338](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304358, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.56.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:45:58.474731 5346 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.336s user 0.111s sys 0.436s
I20260925 02:45:58.475054 5346 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:45:58.475281 5346 mini_ranger_kms.cc:401] 127.5.56.148:42015/kms/v1/keys
Sep 25 02:45:58 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304358, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.56.148@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
Sep 25 02:45:59 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304358, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.56.148@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
Sep 25 02:46:01 dist-test-slave-n563 krb5kdc[16338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:46:01.770965 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:38283
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:38283
--ranger_config_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:42015/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:46:02.161510 16790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:02.161955 16790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:02.162137 16790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:02.169936 16790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:46:02.170137 16790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:02.170271 16790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:02.170384 16790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:46:02.170504 16790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:46:02.181770 16790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.56.148:42015/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:38283
--ranger_config_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:38283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:02.183295 16790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:02.184747 16790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:02.198104 16799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:02.198083 16797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:02.198316 16796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:02.198794 16790 server_base.cc:1034] running on GCE node
I20260925 02:46:02.199751 16790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:02.201357 16790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:02.202589 16790 hybrid_clock.cc:648] HybridClock initialized: now 1790304362202545 us; error 39 us; skew 500 ppm
I20260925 02:46:02.202978 16790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:02 dist-test-slave-n563 krb5kdc[16338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304362, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:02.210446 16790 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:46:02.212059 16790 webserver.cc:460] Webserver started at http://127.5.56.190:36123/ using document root <none> and password file <none>
I20260925 02:46:02.212773 16790 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:02.212903 16790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:46:02.213241 16790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:46:02 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304362, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:02.275885 16790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "b0ec3aaddd174b65b0e26ba62d51f388"
format_stamp: "Formatted at 2026-09-25 02:46:02 on dist-test-slave-n563"
server_key: "c5efd9f2d0e0b24eb184512cb204f3a8"
server_key_iv: "9c7e29b22b5c3806e9fe907a72e7b7df"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:02.277032 16790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "b0ec3aaddd174b65b0e26ba62d51f388"
format_stamp: "Formatted at 2026-09-25 02:46:02 on dist-test-slave-n563"
server_key: "c5efd9f2d0e0b24eb184512cb204f3a8"
server_key_iv: "9c7e29b22b5c3806e9fe907a72e7b7df"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:02.284276 16790 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260925 02:46:02.304970 16806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:02.306869 16790 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:46:02.307087 16790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "b0ec3aaddd174b65b0e26ba62d51f388"
format_stamp: "Formatted at 2026-09-25 02:46:02 on dist-test-slave-n563"
server_key: "c5efd9f2d0e0b24eb184512cb204f3a8"
server_key_iv: "9c7e29b22b5c3806e9fe907a72e7b7df"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:02.307358 16790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:46:02.333377 16790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:02.338632 16790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:02.339002 16790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:02.359733 16790 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:38283
I20260925 02:46:02.359769 16859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:38283 every 8 connection(s)
I20260925 02:46:02.361213 16790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:46:02.365160 16860 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:46:02.366581 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 16790
I20260925 02:46:02.366796 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:46:02.367686 5346 external_mini_cluster.cc:1428] Setting key efc5f3d8faca98649bae7b06982ed982
I20260925 02:46:02.380963 16860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388: Bootstrap starting.
Sep 25 02:46:02 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:46:02.384800 16860 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388: Neither blocks nor log segments found. Creating new log.
I20260925 02:46:02.385957 16860 log.cc:824] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388: Log is configured to *not* fsync() on all Append() calls
I20260925 02:46:02.389026 16860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388: No bootstrap required, opened a new log
I20260925 02:46:02.392062 16863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:02.369911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35152 (local address 127.5.56.190:38283)
0925 02:46:02.370872 (+ 961us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:02.370884 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:02.370927 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:46:02.372541 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:02.372565 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:02.372995 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:02.373376 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:02.375050 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:02.376405 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:02.378260 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:02.379390 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:02.379534 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:02.384173 (+ 4639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:02.384182 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:02.384198 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:02.384241 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:02.387501 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:02.388251 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:02.388259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:02.388265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:02.388428 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:02.388823 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:02.388829 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:02.388842 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:02.389447 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:02.389769 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:02.390213 (+ 444us) server_negotiation.cc:299] Negotiation successful
0925 02:46:02.390650 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":506,"threads_started":1}
I20260925 02:46:02.397707 16860 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ec3aaddd174b65b0e26ba62d51f388" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38283 } }
I20260925 02:46:02.398124 16860 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:46:02.398234 16860 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ec3aaddd174b65b0e26ba62d51f388, State: Initialized, Role: FOLLOWER
I20260925 02:46:02.398913 16860 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ec3aaddd174b65b0e26ba62d51f388" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38283 } }
I20260925 02:46:02.399163 16860 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:46:02.399319 16860 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:46:02.399500 16860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:46:02.402276 16860 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ec3aaddd174b65b0e26ba62d51f388" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38283 } }
I20260925 02:46:02.402982 16860 leader_election.cc:302] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ec3aaddd174b65b0e26ba62d51f388; no voters:
I20260925 02:46:02.403573 16860 leader_election.cc:288] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:46:02.403743 16865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:46:02.405411 16865 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [term 1 LEADER]: Becoming Leader. State: Replica: b0ec3aaddd174b65b0e26ba62d51f388, State: Running, Role: LEADER
I20260925 02:46:02.406105 16860 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:46:02.406018 16865 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ec3aaddd174b65b0e26ba62d51f388" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38283 } }
I20260925 02:46:02.409075 16866 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0ec3aaddd174b65b0e26ba62d51f388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ec3aaddd174b65b0e26ba62d51f388" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38283 } } }
I20260925 02:46:02.409490 16866 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [sys.catalog]: This master's current role is: LEADER
I20260925 02:46:02.410411 16867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ec3aaddd174b65b0e26ba62d51f388. Latest consensus state: current_term: 1 leader_uuid: "b0ec3aaddd174b65b0e26ba62d51f388" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ec3aaddd174b65b0e26ba62d51f388" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 38283 } } }
I20260925 02:46:02.410809 16867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388 [sys.catalog]: This master's current role is: LEADER
I20260925 02:46:02.414059 16872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:46:02.419919 16872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:46:02.420869 16860 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:46:02.438036 16872 catalog_manager.cc:1347] Generated new cluster ID: ca7a917a26d24f7b844b201ac5877454
I20260925 02:46:02.438289 16872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:46:02.452493 16872 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:46:02.454603 16872 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:46:02.471241 16872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388: Generated new TSK 0
I20260925 02:46:02.472500 16872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:46:03.315541 16902 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:46:03.407140 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:42015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38283
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:46:03 dist-test-slave-n563 krb5kdc[16338](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304363, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:46:03.956334 16905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:03.956751 16905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:03.956905 16905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:03.966358 16905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:03.966552 16905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:03.966790 16905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:46:03.977855 16905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.56.148:42015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:03.979336 16905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:03.980782 16905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:04.003782 16917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:04.014128 16919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:04.015004 16915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:04.017585 16905 server_base.cc:1034] running on GCE node
I20260925 02:46:04.018363 16905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:04.019883 16905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:04.021272 16905 hybrid_clock.cc:648] HybridClock initialized: now 1790304364021109 us; error 98 us; skew 500 ppm
I20260925 02:46:04.021765 16905 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:04 dist-test-slave-n563 krb5kdc[16338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:04.025714 16905 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:46:04.039129 16905 webserver.cc:460] Webserver started at http://127.5.56.129:42919/ using document root <none> and password file <none>
I20260925 02:46:04.040086 16905 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:04.040251 16905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:46:04.040689 16905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:46:04 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:04.061419 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "669492180dca4787a90b3cfd75a58683"
format_stamp: "Formatted at 2026-09-25 02:46:04 on dist-test-slave-n563"
server_key: "5bd33ca906d0f341a48bbd9c7ba08803"
server_key_iv: "1859a9761ae83b05fd8a376c4c542c80"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:04.062748 16905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "669492180dca4787a90b3cfd75a58683"
format_stamp: "Formatted at 2026-09-25 02:46:04 on dist-test-slave-n563"
server_key: "5bd33ca906d0f341a48bbd9c7ba08803"
server_key_iv: "1859a9761ae83b05fd8a376c4c542c80"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:04.079192 16905 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.006s sys 0.009s
I20260925 02:46:04.115720 16926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:04.118639 16905 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.003s sys 0.011s
I20260925 02:46:04.118934 16905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "669492180dca4787a90b3cfd75a58683"
format_stamp: "Formatted at 2026-09-25 02:46:04 on dist-test-slave-n563"
server_key: "5bd33ca906d0f341a48bbd9c7ba08803"
server_key_iv: "1859a9761ae83b05fd8a376c4c542c80"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:04.119313 16905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:46:04.174144 16905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:04.180155 16905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:04.180938 16905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:04.188932 16905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:46:04.198161 16905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:46:04.198378 16905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 02:46:04.198623 16905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:46:04.198768 16905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:46:04 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304363, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
Sep 25 02:46:04 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304363, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:04.763617 16905 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:33743
I20260925 02:46:04.763711 17042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:33743 every 8 connection(s)
I20260925 02:46:04.765336 16905 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:46:04.773434 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 16905
I20260925 02:46:04.773720 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:46:04.774834 5346 external_mini_cluster.cc:1428] Setting key 71f916832cfad96b8ea197b6518aa229
Sep 25 02:46:04 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304364, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
I20260925 02:46:04.875375 17046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:04.771617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:41803 (local address 127.5.56.190:38283)
0925 02:46:04.824638 (+ 53021us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:04.824645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:04.830312 (+ 5667us) server_negotiation.cc:407] Connection header received
0925 02:46:04.830418 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:04.830424 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:04.830627 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:04.830770 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:04.839458 (+ 8688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:04.840592 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:04.843343 (+ 2751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:04.844049 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:04.844165 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:04.857109 (+ 12944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:04.857119 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:04.857124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:04.857177 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:04.861093 (+ 3916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:04.872142 (+ 11049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:04.872150 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:04.872154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:04.872377 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:04.873042 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:04.873048 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:04.873052 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:04.873464 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:04.873606 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:04.874684 (+ 1078us) server_negotiation.cc:299] Negotiation successful
0925 02:46:04.874965 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52849,"thread_start_us":53019,"threads_started":1}
I20260925 02:46:04.878183 17045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:04.770349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38283 (local address 127.5.56.129:41803)
0925 02:46:04.825619 (+ 55270us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:04.825696 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:04.829666 (+ 3970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:04.837544 (+ 7878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:04.837558 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:04.838233 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:46:04.839172 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:04.839203 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:04.840859 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:04.840865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:04.843120 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:04.843138 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:04.844563 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:04.844571 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:04.845005 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:04.845997 (+ 992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:46:04.846029 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:46:04.856523 (+ 10494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:46:04.871093 (+ 14570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:04.871105 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:04.871128 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:04.871920 (+ 792us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:04.872582 (+ 662us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:04.872587 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:04.872591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:04.872890 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:04.873710 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:46:04.873720 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:46:04.874183 (+ 463us) client_negotiation.cc:769] Sending connection context
0925 02:46:04.876072 (+ 1889us) client_negotiation.cc:240] Negotiation successful
0925 02:46:04.876698 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54821,"thread_start_us":64392,"threads_started":1}
I20260925 02:46:04.880666 17043 heartbeater.cc:347] Connected to a master server at 127.5.56.190:38283
I20260925 02:46:04.881006 17043 heartbeater.cc:464] Registering TS with master...
I20260925 02:46:04.881948 17043 heartbeater.cc:511] Master 127.5.56.190:38283 requested a full tablet report, sending...
I20260925 02:46:04.884712 16824 ts_manager.cc:194] Registered new tserver with Master: 669492180dca4787a90b3cfd75a58683 (127.5.56.129:33743)
I20260925 02:46:04.886850 16824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:41803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
W20260925 02:46:04.916586 5346 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.5.56.130: real 0.136s user 0.000s sys 0.019s
I20260925 02:46:04.917527 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:42015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38283
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:46:05.543844 17049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:05.544406 17049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:05.544703 17049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:05.557593 17049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:05.557786 17049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:05.557997 17049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:46:05.574442 17049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.56.148:42015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:05.576359 17049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:05.578228 17049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:05.595430 17061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:05.597637 17062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:05.598297 17064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:05.599146 17049 server_base.cc:1034] running on GCE node
I20260925 02:46:05.599995 17049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:05.601311 17049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:05.602555 17049 hybrid_clock.cc:648] HybridClock initialized: now 1790304365602467 us; error 72 us; skew 500 ppm
I20260925 02:46:05.602980 17049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:05 dist-test-slave-n563 krb5kdc[16338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304365, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:05.609225 17049 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:46:05.611658 17049 webserver.cc:460] Webserver started at http://127.5.56.130:43187/ using document root <none> and password file <none>
I20260925 02:46:05.612969 17049 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:05.613262 17049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:46:05.613862 17049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:46:05 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304365, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:05.634820 17049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "989669dc19cb4bfe87909ebdbb0358a0"
format_stamp: "Formatted at 2026-09-25 02:46:05 on dist-test-slave-n563"
server_key: "1063c0ca129a953ddf4c25ef1b614493"
server_key_iv: "e43ddc1c65f86e7a9597977846096c93"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:05.636287 17049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "989669dc19cb4bfe87909ebdbb0358a0"
format_stamp: "Formatted at 2026-09-25 02:46:05 on dist-test-slave-n563"
server_key: "1063c0ca129a953ddf4c25ef1b614493"
server_key_iv: "e43ddc1c65f86e7a9597977846096c93"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:05.644170 17049 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.003s
I20260925 02:46:05.663496 17071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:05.665345 17049 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260925 02:46:05.665619 17049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "989669dc19cb4bfe87909ebdbb0358a0"
format_stamp: "Formatted at 2026-09-25 02:46:05 on dist-test-slave-n563"
server_key: "1063c0ca129a953ddf4c25ef1b614493"
server_key_iv: "e43ddc1c65f86e7a9597977846096c93"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:05.665972 17049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:46:05.691721 17049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:05.696302 17049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:05.696656 17049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:05.697849 17049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:46:05.700193 17049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:46:05.700387 17049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:05.700675 17049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:46:05.700867 17049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:05.771066 17049 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:38025
I20260925 02:46:05.771080 17186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:38025 every 8 connection(s)
I20260925 02:46:05.772902 17049 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:46:05.777522 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 17049
I20260925 02:46:05.777719 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:46:05.778344 5346 external_mini_cluster.cc:1428] Setting key 3a49eae038b0bf17f5660fc5314b6eb9
Sep 25 02:46:05 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304365, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:46:05.802301 17189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:05.776176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:47745 (local address 127.5.56.190:38283)
0925 02:46:05.776855 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:05.776862 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:05.779201 (+ 2339us) server_negotiation.cc:407] Connection header received
0925 02:46:05.781554 (+ 2353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:05.781560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:05.781646 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:05.781777 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:05.783388 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:05.784226 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:05.785979 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:05.786643 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:05.786782 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:05.792858 (+ 6076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:05.792866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:05.792870 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:05.792926 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:05.797477 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:05.798817 (+ 1340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:05.798823 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:05.798827 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:05.799008 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:05.799736 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:05.799741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:05.799745 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:05.800166 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:05.800322 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:05.801393 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0925 02:46:05.801674 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":417,"threads_started":1}
I20260925 02:46:05.803508 17190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:05.776765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38283 (local address 127.5.56.130:47745)
0925 02:46:05.777978 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:05.778025 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:05.781231 (+ 3206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:05.782056 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:05.782065 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:05.782600 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:46:05.783217 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:05.783236 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:05.784404 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:05.784410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:05.785832 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:05.785845 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:05.786938 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:05.786947 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:05.787299 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:05.788298 (+ 999us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:46:05.788335 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:46:05.792498 (+ 4163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:46:05.797676 (+ 5178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:05.797690 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:05.797714 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:05.798643 (+ 929us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:05.799149 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:05.799156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:05.799163 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:05.799581 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:05.800411 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:46:05.800420 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:46:05.800906 (+ 486us) client_negotiation.cc:769] Sending connection context
0925 02:46:05.801238 (+ 332us) client_negotiation.cc:240] Negotiation successful
0925 02:46:05.801855 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":876,"thread_start_us":677,"threads_started":1}
I20260925 02:46:05.805565 17187 heartbeater.cc:347] Connected to a master server at 127.5.56.190:38283
I20260925 02:46:05.805970 17187 heartbeater.cc:464] Registering TS with master...
I20260925 02:46:05.807229 17187 heartbeater.cc:511] Master 127.5.56.190:38283 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
I20260925 02:46:05.809485 16824 ts_manager.cc:194] Registered new tserver with Master: 989669dc19cb4bfe87909ebdbb0358a0 (127.5.56.130:38025)
I20260925 02:46:05.810742 16824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:47745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:46:05.842206 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:42015/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38283
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:46:05.891144 17043 heartbeater.cc:503] Master 127.5.56.190:38283 was elected leader, sending a full tablet report...
W20260925 02:46:06.282727 17193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:06.283174 17193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:06.283430 17193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:06.292681 17193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:06.292863 17193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:06.293047 17193 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:46:06.304349 17193 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.56.148:42015/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:38283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:06.305757 17193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:06.307281 17193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:06.323016 17199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:06.323104 17198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:06.323024 17201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:06.327955 17193 server_base.cc:1034] running on GCE node
I20260925 02:46:06.328771 17193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:06.329815 17193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:06.331051 17193 hybrid_clock.cc:648] HybridClock initialized: now 1790304366330984 us; error 63 us; skew 500 ppm
I20260925 02:46:06.331403 17193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:06 dist-test-slave-n563 krb5kdc[16338](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304366, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:06.334780 17193 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:46:06.336596 17193 webserver.cc:460] Webserver started at http://127.5.56.131:46821/ using document root <none> and password file <none>
I20260925 02:46:06.337339 17193 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:06.337487 17193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:46:06.337836 17193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:46:06 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304366, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:06.354754 17193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "4f3ca887928048f3b3aec45d43c44681"
format_stamp: "Formatted at 2026-09-25 02:46:06 on dist-test-slave-n563"
server_key: "bc5877291d0274674740e015e2037dac"
server_key_iv: "605025551abe2c7933bee7e560d7ee70"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:06.355610 17193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "4f3ca887928048f3b3aec45d43c44681"
format_stamp: "Formatted at 2026-09-25 02:46:06 on dist-test-slave-n563"
server_key: "bc5877291d0274674740e015e2037dac"
server_key_iv: "605025551abe2c7933bee7e560d7ee70"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:06.361703 17193 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260925 02:46:06.381343 17208 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:06.383065 17193 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:46:06.383270 17193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "4f3ca887928048f3b3aec45d43c44681"
format_stamp: "Formatted at 2026-09-25 02:46:06 on dist-test-slave-n563"
server_key: "bc5877291d0274674740e015e2037dac"
server_key_iv: "605025551abe2c7933bee7e560d7ee70"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:06.383517 17193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:46:06.405162 17193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:06.409719 17193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:06.410125 17193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:06.411375 17193 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:46:06.412971 17193 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:46:06.413170 17193 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:06.413360 17193 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:46:06.413450 17193 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:06.460150 17193 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:40965
I20260925 02:46:06.460258 17323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:40965 every 8 connection(s)
I20260925 02:46:06.461675 17193 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:46:06.462922 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 17193
I20260925 02:46:06.463155 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:46:06.463752 5346 external_mini_cluster.cc:1428] Setting key 96725d0337285e4d6d6aca3fc8295786
Sep 25 02:46:06 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304366, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:46:06.483534 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.464868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:52811 (local address 127.5.56.190:38283)
0925 02:46:06.465492 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:06.465498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:06.466534 (+ 1036us) server_negotiation.cc:407] Connection header received
0925 02:46:06.468501 (+ 1967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:06.468506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:06.468573 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:06.468663 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:06.470259 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.471078 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.472773 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.473226 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.473325 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.476871 (+ 3546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:06.476876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:06.476879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:06.476918 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:06.480584 (+ 3666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:06.481280 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:06.481284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:06.481286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:06.481403 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:06.481915 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:06.481918 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:06.481920 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:06.482199 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:06.482311 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:06.483068 (+ 757us) server_negotiation.cc:299] Negotiation successful
0925 02:46:06.483250 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":333,"threads_started":1}
I20260925 02:46:06.484615 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.465354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:38283 (local address 127.5.56.131:52811)
0925 02:46:06.466340 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:06.466425 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:06.468250 (+ 1825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:06.468871 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:06.468879 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:06.469390 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:46:06.470089 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.470107 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.471230 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.471234 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.472650 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.472662 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.473468 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.473474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.473690 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.474352 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:46:06.474435 (+ 83us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:46:06.476654 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:46:06.480707 (+ 4053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:06.480715 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:06.480730 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:06.481162 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:06.481498 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:06.481501 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:06.481503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:06.481819 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:06.482308 (+ 489us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:46:06.482314 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:46:06.482728 (+ 414us) client_negotiation.cc:769] Sending connection context
0925 02:46:06.482972 (+ 244us) client_negotiation.cc:240] Negotiation successful
0925 02:46:06.483375 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":419,"threads_started":1}
I20260925 02:46:06.486512 17324 heartbeater.cc:347] Connected to a master server at 127.5.56.190:38283
I20260925 02:46:06.486904 17324 heartbeater.cc:464] Registering TS with master...
I20260925 02:46:06.487984 17324 heartbeater.cc:511] Master 127.5.56.190:38283 requested a full tablet report, sending...
I20260925 02:46:06.490103 16824 ts_manager.cc:194] Registered new tserver with Master: 4f3ca887928048f3b3aec45d43c44681 (127.5.56.131:40965)
I20260925 02:46:06.491295 16824 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:52811
I20260925 02:46:06.492919 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:46:06.511852 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.498908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35156 (local address 127.5.56.190:38283)
0925 02:46:06.499138 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:06.499145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:06.499401 (+ 256us) server_negotiation.cc:407] Connection header received
0925 02:46:06.499556 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:06.499560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:06.499640 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:06.499728 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:06.501402 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.502112 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.503682 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.504090 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.504186 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.506746 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:06.506751 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:06.506754 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:06.506786 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:06.509479 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:06.510100 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:06.510104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:06.510106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:06.510224 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:06.510619 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:06.510623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:06.510624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:06.510896 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:06.511072 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:06.511303 (+ 231us) server_negotiation.cc:299] Negotiation successful
0925 02:46:06.511486 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260925 02:46:06.516618 16824 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35156:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:46:06.524571 16824 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:46:06.558482 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.537640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:40965 (local address 127.0.0.1:36426)
0925 02:46:06.539602 (+ 1962us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:06.539645 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:06.539831 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:06.549626 (+ 9795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:06.549638 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:06.549685 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:46:06.550371 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.550404 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.553456 (+ 3052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.553464 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.555888 (+ 2424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.555908 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.557432 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.557441 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.557634 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.557691 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 02:46:06.557831 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 02:46:06.558047 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1657,"thread_start_us":12460,"threads_started":1}
I20260925 02:46:06.558507 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.538534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.5.56.131:40965)
0925 02:46:06.548676 (+ 10142us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:06.548691 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:06.548741 (+ 50us) server_negotiation.cc:407] Connection header received
0925 02:46:06.548891 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:06.548903 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:06.549289 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:06.549606 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:46:06.551566 (+ 1960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.553311 (+ 1745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.556051 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.557213 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.557350 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.557602 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:06.557911 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 02:46:06.558102 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9863,"thread_start_us":10246,"threads_started":1}
I20260925 02:46:06.559700 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.537708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:33743 (local address 127.0.0.1:55162)
0925 02:46:06.549180 (+ 11472us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:06.549208 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:06.549360 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:06.551465 (+ 2105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:06.551473 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:06.551510 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:46:06.552102 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.552118 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.554802 (+ 2684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.554810 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.556871 (+ 2061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.556884 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.558907 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.558915 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.559114 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.559156 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:46:06.559248 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 02:46:06.559381 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11297,"thread_start_us":1459,"threads_started":1}
I20260925 02:46:06.560487 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.538128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55162 (local address 127.5.56.129:33743)
0925 02:46:06.550892 (+ 12764us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:06.550898 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:06.550922 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:46:06.551004 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:06.551010 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:06.551195 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:06.551343 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:46:06.552385 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.554614 (+ 2229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.557043 (+ 2429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.558544 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.558689 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.558908 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:06.559973 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0925 02:46:06.560141 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12585,"thread_start_us":12427,"threads_started":1}
I20260925 02:46:06.563162 17257 tablet_service.cc:1467] Processing CreateTablet for tablet 95adaf113327402ba73cdb807a7e5898 (DEFAULT_TABLE table=test-table [id=55aee491cfe748e5931826da0c418e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:46:06.563783 16975 tablet_service.cc:1467] Processing CreateTablet for tablet 95adaf113327402ba73cdb807a7e5898 (DEFAULT_TABLE table=test-table [id=55aee491cfe748e5931826da0c418e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:46:06.564844 17257 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95adaf113327402ba73cdb807a7e5898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:46:06.565493 16975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95adaf113327402ba73cdb807a7e5898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:46:06.565847 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.540058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:38025 (local address 127.0.0.1:40800)
0925 02:46:06.554928 (+ 14870us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:06.554953 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:06.555092 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:06.555809 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:06.555816 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:06.555852 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:46:06.558596 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.558612 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.560478 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.560483 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.562909 (+ 2426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.562924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.565056 (+ 2132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.565062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.565242 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.565277 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 02:46:06.565371 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:46:06.565514 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14705,"thread_start_us":14557,"threads_started":1}
I20260925 02:46:06.566092 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.540337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40800 (local address 127.5.56.130:38025)
0925 02:46:06.554227 (+ 13890us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:06.554236 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:06.555084 (+ 848us) server_negotiation.cc:407] Connection header received
0925 02:46:06.555257 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:06.555269 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:06.555577 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:06.555794 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:46:06.558821 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.560307 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.563087 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.564785 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.565017 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.565235 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:06.565511 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 02:46:06.565763 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13718,"thread_start_us":13856,"threads_started":1}
I20260925 02:46:06.569912 17119 tablet_service.cc:1467] Processing CreateTablet for tablet 95adaf113327402ba73cdb807a7e5898 (DEFAULT_TABLE table=test-table [id=55aee491cfe748e5931826da0c418e16]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:46:06.571748 17119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95adaf113327402ba73cdb807a7e5898. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:46:06.577922 17343 tablet_bootstrap.cc:492] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683: Bootstrap starting.
I20260925 02:46:06.580830 17344 tablet_bootstrap.cc:492] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: Bootstrap starting.
I20260925 02:46:06.584388 17343 tablet_bootstrap.cc:654] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683: Neither blocks nor log segments found. Creating new log.
I20260925 02:46:06.584386 17344 tablet_bootstrap.cc:654] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: Neither blocks nor log segments found. Creating new log.
I20260925 02:46:06.585772 17343 log.cc:824] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683: Log is configured to *not* fsync() on all Append() calls
I20260925 02:46:06.585772 17344 log.cc:824] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: Log is configured to *not* fsync() on all Append() calls
I20260925 02:46:06.586021 17345 tablet_bootstrap.cc:492] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: Bootstrap starting.
I20260925 02:46:06.589361 17345 tablet_bootstrap.cc:654] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: Neither blocks nor log segments found. Creating new log.
I20260925 02:46:06.590564 17345 log.cc:824] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:46:06.594285 17343 tablet_bootstrap.cc:492] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683: No bootstrap required, opened a new log
I20260925 02:46:06.594456 17345 tablet_bootstrap.cc:492] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: No bootstrap required, opened a new log
I20260925 02:46:06.594640 17343 ts_tablet_manager.cc:1397] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.004s
I20260925 02:46:06.594837 17345 ts_tablet_manager.cc:1397] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260925 02:46:06.597083 17344 tablet_bootstrap.cc:492] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: No bootstrap required, opened a new log
I20260925 02:46:06.597458 17344 ts_tablet_manager.cc:1397] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.012s
I20260925 02:46:06.603906 17345 raft_consensus.cc:348] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3ca887928048f3b3aec45d43c44681" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40965 } } peers { permanent_uuid: "669492180dca4787a90b3cfd75a58683" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 33743 } } peers { permanent_uuid: "989669dc19cb4bfe87909ebdbb0358a0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38025 } }
I20260925 02:46:06.604323 17345 raft_consensus.cc:374] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:46:06.604429 17345 raft_consensus.cc:729] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 989669dc19cb4bfe87909ebdbb0358a0, State: Initialized, Role: FOLLOWER
I20260925 02:46:06.605288 17345 consensus_queue.cc:260] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3ca887928048f3b3aec45d43c44681" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40965 } } peers { permanent_uuid: "669492180dca4787a90b3cfd75a58683" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 33743 } } peers { permanent_uuid: "989669dc19cb4bfe87909ebdbb0358a0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38025 } }
I20260925 02:46:06.606694 17343 raft_consensus.cc:348] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3ca887928048f3b3aec45d43c44681" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40965 } } peers { permanent_uuid: "669492180dca4787a90b3cfd75a58683" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 33743 } } peers { permanent_uuid: "989669dc19cb4bfe87909ebdbb0358a0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38025 } }
I20260925 02:46:06.607218 17343 raft_consensus.cc:374] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:46:06.607334 17343 raft_consensus.cc:729] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 669492180dca4787a90b3cfd75a58683, State: Initialized, Role: FOLLOWER
I20260925 02:46:06.607796 17187 heartbeater.cc:503] Master 127.5.56.190:38283 was elected leader, sending a full tablet report...
I20260925 02:46:06.608014 17343 consensus_queue.cc:260] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3ca887928048f3b3aec45d43c44681" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40965 } } peers { permanent_uuid: "669492180dca4787a90b3cfd75a58683" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 33743 } } peers { permanent_uuid: "989669dc19cb4bfe87909ebdbb0358a0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38025 } }
I20260925 02:46:06.608842 17345 ts_tablet_manager.cc:1428] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260925 02:46:06.609671 17344 raft_consensus.cc:348] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3ca887928048f3b3aec45d43c44681" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40965 } } peers { permanent_uuid: "669492180dca4787a90b3cfd75a58683" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 33743 } } peers { permanent_uuid: "989669dc19cb4bfe87909ebdbb0358a0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38025 } }
I20260925 02:46:06.610113 17344 raft_consensus.cc:374] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:46:06.610208 17344 raft_consensus.cc:729] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3ca887928048f3b3aec45d43c44681, State: Initialized, Role: FOLLOWER
I20260925 02:46:06.610497 17343 ts_tablet_manager.cc:1428] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
I20260925 02:46:06.610994 17344 consensus_queue.cc:260] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3ca887928048f3b3aec45d43c44681" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40965 } } peers { permanent_uuid: "669492180dca4787a90b3cfd75a58683" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 33743 } } peers { permanent_uuid: "989669dc19cb4bfe87909ebdbb0358a0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38025 } }
I20260925 02:46:06.612191 17324 heartbeater.cc:503] Master 127.5.56.190:38283 was elected leader, sending a full tablet report...
I20260925 02:46:06.612905 17344 ts_tablet_manager.cc:1428] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260925 02:46:06.690573 17350 raft_consensus.cc:482] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:46:06.690961 17350 raft_consensus.cc:504] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3ca887928048f3b3aec45d43c44681" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40965 } } peers { permanent_uuid: "669492180dca4787a90b3cfd75a58683" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 33743 } } peers { permanent_uuid: "989669dc19cb4bfe87909ebdbb0358a0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38025 } }
I20260925 02:46:06.692780 17350 leader_election.cc:288] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4f3ca887928048f3b3aec45d43c44681 (127.5.56.131:40965), 989669dc19cb4bfe87909ebdbb0358a0 (127.5.56.130:38025)
I20260925 02:46:06.700770 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.693267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:59901 (local address 127.5.56.130:38025)
0925 02:46:06.693506 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:06.693513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:06.694233 (+ 720us) server_negotiation.cc:407] Connection header received
0925 02:46:06.694497 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:06.694503 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:06.694573 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:06.694675 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:46:06.695287 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.696310 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.698558 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.699701 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.699837 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.699892 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:06.700375 (+ 483us) server_negotiation.cc:299] Negotiation successful
0925 02:46:06.700471 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 02:46:06.701063 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.693109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:38025 (local address 127.5.56.129:59901)
0925 02:46:06.694108 (+ 999us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:06.694125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:06.694308 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:06.694700 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:06.694704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:06.694732 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:46:06.695117 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.695129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.696450 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.696455 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.698336 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.698347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.699941 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.699948 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.700160 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.700197 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:46:06.700343 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 02:46:06.700506 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":32,"thread_start_us":709,"threads_started":1}
I20260925 02:46:06.701524 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.693255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:45855 (local address 127.5.56.131:40965)
0925 02:46:06.693506 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:06.693513 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:06.694803 (+ 1290us) server_negotiation.cc:407] Connection header received
0925 02:46:06.695069 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:06.695073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:06.695129 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:06.695231 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:46:06.695878 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.697009 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.699161 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.700462 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.700642 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.700736 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:06.701153 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 02:46:06.701275 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260925 02:46:06.701611 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.693582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:40965 (local address 127.5.56.129:45855)
0925 02:46:06.694693 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:06.694709 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:06.694896 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:06.695250 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:06.695254 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:06.695276 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:46:06.695725 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.695737 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.697144 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.697148 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.699002 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:06.699018 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.700695 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.700700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:06.700916 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.700957 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:46:06.701109 (+ 152us) client_negotiation.cc:240] Negotiation successful
0925 02:46:06.701240 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":995,"mutex_wait_us":3,"thread_start_us":813,"threads_started":1}
I20260925 02:46:06.702569 17142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95adaf113327402ba73cdb807a7e5898" candidate_uuid: "669492180dca4787a90b3cfd75a58683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "989669dc19cb4bfe87909ebdbb0358a0" is_pre_election: true
I20260925 02:46:06.702867 17278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95adaf113327402ba73cdb807a7e5898" candidate_uuid: "669492180dca4787a90b3cfd75a58683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3ca887928048f3b3aec45d43c44681" is_pre_election: true
I20260925 02:46:06.703562 17142 raft_consensus.cc:2454] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 669492180dca4787a90b3cfd75a58683 in term 0.
I20260925 02:46:06.703641 17278 raft_consensus.cc:2454] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 669492180dca4787a90b3cfd75a58683 in term 0.
I20260925 02:46:06.704623 16927 leader_election.cc:302] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 669492180dca4787a90b3cfd75a58683, 989669dc19cb4bfe87909ebdbb0358a0; no voters:
I20260925 02:46:06.705068 17350 raft_consensus.cc:2781] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:46:06.705219 17350 raft_consensus.cc:482] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:46:06.705345 17350 raft_consensus.cc:3037] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:46:06.707911 17350 raft_consensus.cc:504] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3ca887928048f3b3aec45d43c44681" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40965 } } peers { permanent_uuid: "669492180dca4787a90b3cfd75a58683" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 33743 } } peers { permanent_uuid: "989669dc19cb4bfe87909ebdbb0358a0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38025 } }
I20260925 02:46:06.708652 17350 leader_election.cc:288] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [CANDIDATE]: Term 1 election: Requested vote from peers 4f3ca887928048f3b3aec45d43c44681 (127.5.56.131:40965), 989669dc19cb4bfe87909ebdbb0358a0 (127.5.56.130:38025)
I20260925 02:46:06.709129 17278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95adaf113327402ba73cdb807a7e5898" candidate_uuid: "669492180dca4787a90b3cfd75a58683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3ca887928048f3b3aec45d43c44681"
I20260925 02:46:06.709204 17142 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95adaf113327402ba73cdb807a7e5898" candidate_uuid: "669492180dca4787a90b3cfd75a58683" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "989669dc19cb4bfe87909ebdbb0358a0"
I20260925 02:46:06.709334 17278 raft_consensus.cc:3037] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:46:06.709463 17142 raft_consensus.cc:3037] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:46:06.711748 17278 raft_consensus.cc:2454] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 669492180dca4787a90b3cfd75a58683 in term 1.
I20260925 02:46:06.712188 16931 leader_election.cc:302] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f3ca887928048f3b3aec45d43c44681, 669492180dca4787a90b3cfd75a58683; no voters:
I20260925 02:46:06.712477 17142 raft_consensus.cc:2454] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 669492180dca4787a90b3cfd75a58683 in term 1.
I20260925 02:46:06.712567 17350 raft_consensus.cc:2781] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:46:06.713762 17350 raft_consensus.cc:686] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 1 LEADER]: Becoming Leader. State: Replica: 669492180dca4787a90b3cfd75a58683, State: Running, Role: LEADER
W20260925 02:46:06.714334 17325 tablet.cc:2369] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:46:06.714520 17350 consensus_queue.cc:237] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3ca887928048f3b3aec45d43c44681" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40965 } } peers { permanent_uuid: "669492180dca4787a90b3cfd75a58683" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 33743 } } peers { permanent_uuid: "989669dc19cb4bfe87909ebdbb0358a0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38025 } }
I20260925 02:46:06.719935 16822 catalog_manager.cc:5495] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 reported cstate change: term changed from 0 to 1, leader changed from <none> to 669492180dca4787a90b3cfd75a58683 (127.5.56.129). New cstate: current_term: 1 leader_uuid: "669492180dca4787a90b3cfd75a58683" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4f3ca887928048f3b3aec45d43c44681" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "669492180dca4787a90b3cfd75a58683" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 33743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "989669dc19cb4bfe87909ebdbb0358a0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38025 } health_report { overall_health: UNKNOWN } } }
I20260925 02:46:06.746209 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.740949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55170 (local address 127.5.56.129:33743)
0925 02:46:06.741187 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:06.741192 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:06.741210 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:46:06.741306 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:06.741314 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:06.741405 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:06.741536 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:46:06.741880 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.742683 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.744122 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.744610 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.744706 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.745171 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:46:06.745684 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:46:06.745791 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:06.745907 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 02:46:06.745977 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 02:46:06.755131 17142 raft_consensus.cc:1261] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0 [term 1 FOLLOWER]: Refusing update from remote peer 669492180dca4787a90b3cfd75a58683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:46:06.755131 17278 raft_consensus.cc:1261] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681 [term 1 FOLLOWER]: Refusing update from remote peer 669492180dca4787a90b3cfd75a58683: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:46:06.756269 17354 consensus_queue.cc:1035] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "989669dc19cb4bfe87909ebdbb0358a0" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:46:06.756719 17350 consensus_queue.cc:1035] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f3ca887928048f3b3aec45d43c44681" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:46:06.775787 17188 tablet.cc:2369] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:46:06.785171 17357 mvcc.cc:205] Tried to move back new op lower bound from 7333086686217781248 to 7333086686066864128. Current Snapshot: MvccSnapshot[applied={T|T < 7333086686217781248}]
I20260925 02:46:06.787535 17358 mvcc.cc:205] Tried to move back new op lower bound from 7333086686217781248 to 7333086686066864128. Current Snapshot: MvccSnapshot[applied={T|T < 7333086686217781248}]
I20260925 02:46:06.794822 17359 mvcc.cc:205] Tried to move back new op lower bound from 7333086686217781248 to 7333086686066864128. Current Snapshot: MvccSnapshot[applied={T|T < 7333086686217781248}]
I20260925 02:46:06.811909 16822 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:46:06.812222 16822 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35156:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:46:06.816376 16822 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0ec3aaddd174b65b0e26ba62d51f388: Sending DeleteTablet for 3 replicas of tablet 95adaf113327402ba73cdb807a7e5898
I20260925 02:46:06.817747 17257 tablet_service.cc:1514] Processing DeleteTablet for tablet 95adaf113327402ba73cdb807a7e5898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:46:06 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:36426
I20260925 02:46:06.817943 17119 tablet_service.cc:1514] Processing DeleteTablet for tablet 95adaf113327402ba73cdb807a7e5898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:46:06 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:40800
I20260925 02:46:06.818048 16975 tablet_service.cc:1514] Processing DeleteTablet for tablet 95adaf113327402ba73cdb807a7e5898 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:46:06 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:55162
I20260925 02:46:06.819231 17369 tablet_replica.cc:330] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: stopping tablet replica
I20260925 02:46:06.819784 17369 raft_consensus.cc:2229] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:46:06.820333 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 16905
I20260925 02:46:06.820513 17369 raft_consensus.cc:2258] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:46:06.823791 17369 ts_tablet_manager.cc:1905] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:46:06.830607 17370 tablet_replica.cc:330] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683: stopping tablet replica
I20260925 02:46:06.831275 17370 raft_consensus.cc:2229] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:46:06.832379 17370 raft_consensus.cc:2258] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:46:06.832582 17369 ts_tablet_manager.cc:1918] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:46:06.832801 17369 log.cc:1196] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/95adaf113327402ba73cdb807a7e5898
I20260925 02:46:06.833434 17369 ts_tablet_manager.cc:1939] T 95adaf113327402ba73cdb807a7e5898 P 989669dc19cb4bfe87909ebdbb0358a0: Deleting consensus metadata
I20260925 02:46:06.834694 17370 ts_tablet_manager.cc:1905] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:46:06.835270 17368 tablet_replica.cc:330] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: stopping tablet replica
I20260925 02:46:06.835754 17368 raft_consensus.cc:2229] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:46:06.835863 16807 catalog_manager.cc:4841] TS 989669dc19cb4bfe87909ebdbb0358a0 (127.5.56.130:38025): tablet 95adaf113327402ba73cdb807a7e5898 (table test-table [id=55aee491cfe748e5931826da0c418e16]) successfully deleted
I20260925 02:46:06.836287 17368 raft_consensus.cc:2258] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:46:06.838032 17368 ts_tablet_manager.cc:1905] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:46:06.842907 17370 ts_tablet_manager.cc:1918] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:46:06.843087 17370 log.cc:1196] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/95adaf113327402ba73cdb807a7e5898
I20260925 02:46:06.843693 17370 ts_tablet_manager.cc:1939] T 95adaf113327402ba73cdb807a7e5898 P 669492180dca4787a90b3cfd75a58683: Deleting consensus metadata
I20260925 02:46:06.845410 17368 ts_tablet_manager.cc:1918] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:46:06.845610 17368 log.cc:1196] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/95adaf113327402ba73cdb807a7e5898
I20260925 02:46:06.845852 16807 catalog_manager.cc:4841] TS 669492180dca4787a90b3cfd75a58683 (127.5.56.129:33743): tablet 95adaf113327402ba73cdb807a7e5898 (table test-table [id=55aee491cfe748e5931826da0c418e16]) successfully deleted
I20260925 02:46:06.846240 17368 ts_tablet_manager.cc:1939] T 95adaf113327402ba73cdb807a7e5898 P 4f3ca887928048f3b3aec45d43c44681: Deleting consensus metadata
Sep 25 02:46:06 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:46:06.847895 16810 catalog_manager.cc:4841] TS 4f3ca887928048f3b3aec45d43c44681 (127.5.56.131:40965): tablet 95adaf113327402ba73cdb807a7e5898 (table test-table [id=55aee491cfe748e5931826da0c418e16]) successfully deleted
I20260925 02:46:06.855468 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.821237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55186 (local address 127.5.56.129:33743)
0925 02:46:06.821460 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:06.821470 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:06.827027 (+ 5557us) server_negotiation.cc:407] Connection header received
0925 02:46:06.828206 (+ 1179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:06.828215 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:06.828325 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:06.828428 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:06.832840 (+ 4412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.834171 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.840379 (+ 6208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.841171 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.841294 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.848470 (+ 7176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:06.848483 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:06.848498 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:06.848535 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:06.852257 (+ 3722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:06.853225 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:06.853233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:06.853235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:06.853409 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:06.853911 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:06.853915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:06.853917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:06.854297 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:06.854636 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:06.854947 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 02:46:06.855144 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 02:46:06.856292 17038 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:46:06.958673 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 16905
I20260925 02:46:06.978691 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 17049
Sep 25 02:46:06 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:46:06.995661 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:06.979714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40808 (local address 127.5.56.130:38025)
0925 02:46:06.979934 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:06.979940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:06.979956 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:46:06.980007 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:06.980011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:06.980091 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:06.980218 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:06.981747 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.982805 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.984161 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:06.984701 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:06.984860 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:06.989601 (+ 4741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:06.989613 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:06.989628 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:06.989665 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:06.992748 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:06.993519 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:06.993525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:06.993528 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:06.993658 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:06.994006 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:06.994009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:06.994011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:06.994579 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:06.994806 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:06.995162 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 02:46:06.995369 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 02:46:06.996407 17182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:46:07.087734 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 17049
I20260925 02:46:07.105340 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 17193
Sep 25 02:46:07 dist-test-slave-n563 krb5kdc[16338](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:46:07.121517 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:07.106225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36438 (local address 127.5.56.131:40965)
0925 02:46:07.106435 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:07.106442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:07.106489 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:46:07.106642 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:07.106646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:07.106727 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:07.106812 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:07.108228 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:07.108963 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:07.110226 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:07.110743 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:07.110844 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:07.115869 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:07.115887 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:07.115923 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:07.115962 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:07.118948 (+ 2986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:07.119668 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:07.119675 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:07.119678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:07.119806 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:07.120150 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:07.120153 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:07.120155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:07.120528 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:07.120744 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:07.121076 (+ 332us) server_negotiation.cc:299] Negotiation successful
0925 02:46:07.121248 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:46:07.122169 17319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:46:07.214143 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 17193
I20260925 02:46:07.235432 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 16790
I20260925 02:46:07.236688 16855 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:46:07.358875 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 16790
I20260925 02:46:07.381856 5346 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:46:07.718526 5346 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:46:07.718885 5346 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:46:08.065042 5346 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:46:08.065 UTC [16369] LOG: received smart shutdown request
2026-09-25 02:46:08.069 UTC [16369] LOG: background worker "logical replication launcher" (PID 16378) exited with exit code 1
2026-09-25 02:46:08.070 UTC [16373] LOG: shutting down
2026-09-25 02:46:08.256 UTC [16369] LOG: database system is shut down
2026-09-25T02:46:08Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51796 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:46:08 dist-test-slave-n563 krb5kdc[17384](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:46:08 dist-test-slave-n563 krb5kdc[17384](info): set up 2 sockets
Sep 25 02:46:08 dist-test-slave-n563 krb5kdc[17384](info): commencing operation
krb5kdc: starting...
W20260925 02:46:09.512742 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:46:09 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304369, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:46:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:46:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.5.56.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.5.56.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/rangeradmin_127.5.56.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/rangeradmin_127.5.56.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.5.56.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.5.56.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/rangerlookup_127.5.56.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/rangerlookup_127.5.56.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.56.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.5.56.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/HTTP_127.5.56.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/HTTP_127.5.56.148@KRBTEST.COM.keytab.
I20260925 02:46:09.860848 5346 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task87C_4R/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/postgres -l logfile start
2026-09-25 02:46:12.414 UTC [17414] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 02:46:12.414 UTC [17414] LOG: listening on IPv4 address "127.5.56.148", port 44189
2026-09-25 02:46:12.417 UTC [17414] LOG: listening on Unix socket "/tmp/.s.PGSQL.44189"
2026-09-25 02:46:12.440 UTC [17417] LOG: database system was shut down at 2026-09-25 02:46:10 UTC
2026-09-25 02:46:12.444 UTC [17414] LOG: database system is ready to accept connections
I20260925 02:46:13.435758 5346 mini_postgres.cc:96] Postgres bound to 44189
2026-09-25 02:46:13.454 UTC [17426] FATAL: database "slave" does not exist
127.5.56.148:44189 - accepting connections
I20260925 02:46:13.455222 5346 mini_ranger.cc:161] Starting Ranger...
I20260925 02:46:13.483592 5346 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 02:46:13.971880 5346 mini_ranger.cc:87] Created ranger Postgres database
I20260925 02:46:13.972021 5346 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-admin
2026-09-25 02:46:14,249 [I] DB FLAVOR :POSTGRES
2026-09-25 02:46:14,250 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:46:14,250 [I] Checking connection..
2026-09-25 02:46:14,250 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 02:46:14,496 [I] Checking connection passed.
2026-09-25 02:46:14,496 [I] --------- Verifying version history table ---------
2026-09-25 02:46:14,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:46:14,741 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:46:14,741 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:46:14,988 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 02:46:14,988 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 02:46:14,989 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 02:46:15.254 UTC [17490] WARNING: there is no transaction in progress
2026-09-25 02:46:15,260 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 02:46:15,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 02:46:15,522 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 02:46:15,523 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:46:15,523 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 02:46:15,757 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 02:46:16,012 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-n563.c.gcp-upstream.internal','N') ;"
2026-09-25 02:46:16,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:46:16,542 [I] Table x_portal_user does not exist in database ranger
2026-09-25 02:46:16,542 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:46:16,812 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 02:46:16,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:46:17,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:46:17,332 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 02:46:17,332 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 02:46:18.308 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.325 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.342 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.359 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.376 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.395 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.458 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.464 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.479 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.489 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.500 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.513 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.523 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.531 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.538 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:18.546 UTC [17601] WARNING: there is no transaction in progress
2026-09-25 02:46:19,212 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 02:46:19,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 02:46:19,460 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 02:46:19,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 02:46:19,712 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 02:46:19,713 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 02:46:19,971 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 02:46:19,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 02:46:20,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 02:46:20,509 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-n563.c.gcp-upstream.internal';"
2026-09-25 02:46:20,751 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 02:46:20,752 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 02:46:21,011 [I] DB_PATCHES have already been applied
I20260925 02:46:21.017274 5346 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-admin:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task87C_4R/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 02:46:21.017366 5346 mini_ranger.cc:193] Using host: 127.5.56.148
I20260925 02:46:21.033402 5346 mini_ranger.cc:227] Ranger admin URL: http://127.5.56.148:41415
Sep 25, 2026 2:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:46:21 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:46:22 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 02:46:22.269819 5346 mini_ranger.cc:160] Time spent starting Ranger: real 8.815s user 0.000s sys 0.044s
Sep 25, 2026 2:46:41 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:46:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 25, 2026 2:46:41 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:46:42 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:46:43 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 02:46:43.611503 5346 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.5.56.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.5.56.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/rangerkms_127.5.56.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.5.56.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/rangerkms_127.5.56.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.5.56.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.5.56.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.5.56.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/HTTP_127.5.56.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.5.56.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/HTTP_127.5.56.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 02:46:43 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304403, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 02:46:43.785552 5346 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 02:46:43.808288 5346 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 02:46:44.658108 5346 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 02:46:44.658226 5346 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-kms
I20260925 02:46:44.658278 5346 mini_ranger_kms.cc:227] Using postgres at 127.5.56.148:44189
2026-09-25 02:46:45,038 [I] DB FLAVOR :POSTGRES
2026-09-25 02:46:45,039 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 02:46:45,039 [I] Checking connection
2026-09-25 02:46:45,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 02:46:45,294 [I] connection success
2026-09-25 02:46:45,294 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 02:46:45,295 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 02:46:45,295 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 02:46:45,531 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 02:46:45,531 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 02:46:45,532 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 02:46:45,532 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.5.56.148:44189/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 02:46:45,792 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 02:46:46.250618 5346 mini_ranger_kms.cc:310] Created kms service
I20260925 02:46:46.462854 5346 mini_ranger_kms.cc:326] Created kudu user
I20260925 02:46:46.508006 5346 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 02:46:46 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304403, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:46.893389 5346 mini_ranger_kms.cc:384] Added ranger policy
I20260925 02:46:46.893697 5346 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-kms:/tmp/dist-test-task87C_4R/build/asan/bin/postgresql.jar:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task87C_4R/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 02:46:46.893754 5346 mini_ranger_kms.cc:240] Using host: 127.5.56.148
I20260925 02:46:46.916282 5346 mini_ranger_kms.cc:276] Ranger KMS PID: 17784
I20260925 02:46:46.916425 5346 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.5.56.148:48043
Sep 25, 2026 2:46:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:46:48 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:46:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:46:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:46:48 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:46:48 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 02:46:49.245438 5346 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.460s user 0.063s sys 0.337s
I20260925 02:46:49.245708 5346 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 02:46:49.245891 5346 mini_ranger_kms.cc:401] 127.5.56.148:48043/kms/v1/keys
Sep 25 02:46:49 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.56.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 02:46:49 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.56.148@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
Sep 25 02:46:50 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304409, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.5.56.148@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
Sep 25 02:46:52 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:46:52.588351 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:42991
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:42991
--ranger_config_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:48043/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:46:52.935348 17835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:52.935735 17835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:52.935897 17835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:52.944818 17835 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:46:52.944973 17835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:52.945055 17835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:52.945135 17835 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:46:52.945215 17835 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:46:52.955832 17835 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal
--ranger_kms_url=127.5.56.148:48043/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:42991
--ranger_config_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:42991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:52.957216 17835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:52.958655 17835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:52.970834 17841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:52.970969 17840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:52.970948 17843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:52.971638 17835 server_base.cc:1034] running on GCE node
I20260925 02:46:52.972441 17835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:52.973948 17835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:52.975134 17835 hybrid_clock.cc:648] HybridClock initialized: now 1790304412975085 us; error 42 us; skew 500 ppm
I20260925 02:46:52.975446 17835 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:52 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304412, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:52.983440 17835 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:46:52.985169 17835 webserver.cc:460] Webserver started at http://127.5.56.190:35021/ using document root <none> and password file <none>
I20260925 02:46:52.985870 17835 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:52.986001 17835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:46:52.986335 17835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:46:52 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304412, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:53.046878 17835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "605bc8465e79412399a52e2fbd678dcd"
format_stamp: "Formatted at 2026-09-25 02:46:53 on dist-test-slave-n563"
server_key: "0203859277a20bd98df62840ff2cb741"
server_key_iv: "f46954b56c4355b923aa6dd9156a46b4"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:53.048101 17835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "605bc8465e79412399a52e2fbd678dcd"
format_stamp: "Formatted at 2026-09-25 02:46:53 on dist-test-slave-n563"
server_key: "0203859277a20bd98df62840ff2cb741"
server_key_iv: "f46954b56c4355b923aa6dd9156a46b4"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:53.055379 17835 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260925 02:46:53.075757 17850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:53.077606 17835 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:46:53.077821 17835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "605bc8465e79412399a52e2fbd678dcd"
format_stamp: "Formatted at 2026-09-25 02:46:53 on dist-test-slave-n563"
server_key: "0203859277a20bd98df62840ff2cb741"
server_key_iv: "f46954b56c4355b923aa6dd9156a46b4"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:53.078084 17835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:46:53.103785 17835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:53.108597 17835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:53.108952 17835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:53.128473 17835 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:42991
I20260925 02:46:53.128561 17902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:42991 every 8 connection(s)
I20260925 02:46:53.129984 17835 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:46:53.134727 17903 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:46:53.138087 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 17835
I20260925 02:46:53.138324 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:46:53.139083 5346 external_mini_cluster.cc:1428] Setting key 2829afb85d8821f3a7dc026ad5069d6b
I20260925 02:46:53.149276 17903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd: Bootstrap starting.
I20260925 02:46:53.153060 17903 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd: Neither blocks nor log segments found. Creating new log.
I20260925 02:46:53.154553 17903 log.cc:824] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd: Log is configured to *not* fsync() on all Append() calls
I20260925 02:46:53.162729 17903 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd: No bootstrap required, opened a new log
Sep 25 02:46:53 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:46:53.172745 17903 raft_consensus.cc:348] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } }
I20260925 02:46:53.173403 17903 raft_consensus.cc:374] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:46:53.173520 17903 raft_consensus.cc:729] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605bc8465e79412399a52e2fbd678dcd, State: Initialized, Role: FOLLOWER
I20260925 02:46:53.174284 17903 consensus_queue.cc:260] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } }
I20260925 02:46:53.174623 17903 raft_consensus.cc:388] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:46:53.174777 17903 raft_consensus.cc:482] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:46:53.174980 17903 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:46:53.177934 17903 raft_consensus.cc:504] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } }
I20260925 02:46:53.178617 17903 leader_election.cc:302] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 605bc8465e79412399a52e2fbd678dcd; no voters:
I20260925 02:46:53.180341 17906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:53.141565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49680 (local address 127.5.56.190:42991)
0925 02:46:53.154020 (+ 12455us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:53.154037 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:53.154091 (+ 54us) server_negotiation.cc:407] Connection header received
0925 02:46:53.157453 (+ 3362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:53.157498 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:53.158223 (+ 725us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:53.158836 (+ 613us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:53.160477 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:53.162449 (+ 1972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:53.164318 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:53.165386 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:53.165575 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:53.170835 (+ 5260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:53.170847 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:53.170874 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:53.170933 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:53.175014 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:53.175895 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:53.175905 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:53.175915 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:53.176105 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:53.176539 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:53.176544 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:53.176548 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:53.177304 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:53.177570 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:53.178118 (+ 548us) server_negotiation.cc:299] Negotiation successful
0925 02:46:53.178589 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12199,"thread_start_us":11922,"threads_started":1}
I20260925 02:46:53.181125 17903 leader_election.cc:288] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:46:53.181238 17908 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:46:53.186093 17908 raft_consensus.cc:686] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 1 LEADER]: Becoming Leader. State: Replica: 605bc8465e79412399a52e2fbd678dcd, State: Running, Role: LEADER
I20260925 02:46:53.186851 17908 consensus_queue.cc:237] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } }
I20260925 02:46:53.187461 17903 sys_catalog.cc:564] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:46:53.190587 17910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 605bc8465e79412399a52e2fbd678dcd. Latest consensus state: current_term: 1 leader_uuid: "605bc8465e79412399a52e2fbd678dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } } }
I20260925 02:46:53.190971 17910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [sys.catalog]: This master's current role is: LEADER
I20260925 02:46:53.190135 17909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "605bc8465e79412399a52e2fbd678dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } } }
I20260925 02:46:53.191671 17909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [sys.catalog]: This master's current role is: LEADER
I20260925 02:46:53.204079 17913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:46:53.211354 17913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:46:53.262373 17913 catalog_manager.cc:1347] Generated new cluster ID: 0fd90f3ea38445239483995cda65d0c3
I20260925 02:46:53.262550 17913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:46:53.266129 17903 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 02:46:53.286471 17913 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:46:53.289295 17913 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:46:53.299408 17913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd: Generated new TSK 0
I20260925 02:46:53.300382 17913 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:46:54.070992 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:48043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42991
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:46:54 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304414, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 02:46:54.545826 17948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:54.546259 17948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:54.546476 17948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:54.554816 17948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:54.554939 17948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:54.555119 17948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:46:54.565397 17948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.5.56.148:48043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:54.566833 17948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:54.568459 17948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:54.588598 17959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:54.591110 17960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:54.598613 17962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:54.599429 17948 server_base.cc:1034] running on GCE node
I20260925 02:46:54.600317 17948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:54.605741 17948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:54.607113 17948 hybrid_clock.cc:648] HybridClock initialized: now 1790304414606999 us; error 112 us; skew 500 ppm
I20260925 02:46:54.607613 17948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:54 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304414, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:54.612232 17948 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:46:54.615005 17948 webserver.cc:460] Webserver started at http://127.5.56.129:45843/ using document root <none> and password file <none>
I20260925 02:46:54.616026 17948 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:54.616204 17948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:46:54.616570 17948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:46:54 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304414, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:54.639266 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "97c83d8b2dde4fa4a844a00e6def068a"
format_stamp: "Formatted at 2026-09-25 02:46:54 on dist-test-slave-n563"
server_key: "4b799560b63925317806a34e3248ef59"
server_key_iv: "3ce038a59fb405b70394ab12862f4c8f"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:54.640525 17948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "97c83d8b2dde4fa4a844a00e6def068a"
format_stamp: "Formatted at 2026-09-25 02:46:54 on dist-test-slave-n563"
server_key: "4b799560b63925317806a34e3248ef59"
server_key_iv: "3ce038a59fb405b70394ab12862f4c8f"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:54.648432 17948 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260925 02:46:54.675114 17969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:54.677408 17948 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.007s sys 0.001s
I20260925 02:46:54.677711 17948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "97c83d8b2dde4fa4a844a00e6def068a"
format_stamp: "Formatted at 2026-09-25 02:46:54 on dist-test-slave-n563"
server_key: "4b799560b63925317806a34e3248ef59"
server_key_iv: "3ce038a59fb405b70394ab12862f4c8f"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:54.678087 17948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:46:54.718183 17948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:54.724545 17948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:54.724968 17948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:54.726366 17948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:46:54.751459 17948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:46:54.751637 17948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260925 02:46:54.751829 17948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:46:54.751929 17948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 02:46:54 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304414, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:54.901150 17948 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:46811
I20260925 02:46:54.901168 18082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:46811 every 8 connection(s)
I20260925 02:46:54.902971 17948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:46:54.912014 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 17948
I20260925 02:46:54.912230 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:46:54.912887 5346 external_mini_cluster.cc:1428] Setting key 6153bf4a9c130f1b522c89641862c573
Sep 25 02:46:54 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304414, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:46:54.939306 18085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:54.905868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:35687 (local address 127.5.56.190:42991)
0925 02:46:54.906827 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:54.906835 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:54.911266 (+ 4431us) server_negotiation.cc:407] Connection header received
0925 02:46:54.913853 (+ 2587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:54.913860 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:54.913966 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:54.914083 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:54.916032 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:54.917197 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:54.919991 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:54.920777 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:54.920937 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:54.930330 (+ 9393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:54.930341 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:54.930349 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:54.930429 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:54.934631 (+ 4202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:54.935858 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:54.935868 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:54.935874 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:54.936046 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:54.936639 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:54.936647 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:54.936652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:54.937083 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:54.937233 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:54.938465 (+ 1232us) server_negotiation.cc:299] Negotiation successful
0925 02:46:54.938771 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":644,"threads_started":1}
I20260925 02:46:54.942237 18086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:54.906249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42991 (local address 127.5.56.129:35687)
0925 02:46:54.911044 (+ 4795us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:54.911107 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:54.913478 (+ 2371us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:54.914358 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:54.914367 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:54.914970 (+ 603us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:46:54.915802 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:54.915844 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:54.917378 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:54.917387 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:54.919798 (+ 2411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:54.919823 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:54.921043 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:54.921052 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:54.921413 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:54.926551 (+ 5138us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:46:54.926593 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:46:54.929976 (+ 3383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:46:54.934906 (+ 4930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:54.934920 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:54.934946 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:54.935693 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:54.936212 (+ 519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:54.936220 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:54.936226 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:54.936503 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:54.937349 (+ 846us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:46:54.937360 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:46:54.937924 (+ 564us) client_negotiation.cc:769] Sending connection context
0925 02:46:54.939941 (+ 2017us) client_negotiation.cc:240] Negotiation successful
0925 02:46:54.940581 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4479,"thread_start_us":4274,"threads_started":1}
I20260925 02:46:54.945734 18083 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42991
I20260925 02:46:54.946098 18083 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
I20260925 02:46:54.954147 18083 heartbeater.cc:511] Master 127.5.56.190:42991 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
I20260925 02:46:54.958122 17867 ts_manager.cc:194] Registered new tserver with Master: 97c83d8b2dde4fa4a844a00e6def068a (127.5.56.129:46811)
I20260925 02:46:54.961701 17867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:35687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:46:55.000233 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:48043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42991
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:46:55 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304414, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
W20260925 02:46:55.447721 18089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:55.448012 18089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:55.448199 18089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:55.456025 18089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:55.456174 18089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:55.456367 18089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:46:55.466979 18089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.5.56.148:48043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:55.468372 18089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:55.469709 18089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:55.483853 18104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:55.483888 18106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:55.483832 18103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:55.484642 18089 server_base.cc:1034] running on GCE node
I20260925 02:46:55.485343 18089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:55.486322 18089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:55.487574 18089 hybrid_clock.cc:648] HybridClock initialized: now 1790304415487512 us; error 65 us; skew 500 ppm
I20260925 02:46:55.487906 18089 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:55 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304415, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:55.491261 18089 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:46:55.492897 18089 webserver.cc:460] Webserver started at http://127.5.56.130:42523/ using document root <none> and password file <none>
I20260925 02:46:55.493700 18089 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:55.493817 18089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:46:55.494172 18089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:46:55 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304415, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:55.510696 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "eb42d44fb6de4b34a056e8762ec49eb3"
format_stamp: "Formatted at 2026-09-25 02:46:55 on dist-test-slave-n563"
server_key: "d6b10301246533dd1471b375a470c30c"
server_key_iv: "7087cef70c99b9eb950fb82d6f978745"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:55.511607 18089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "eb42d44fb6de4b34a056e8762ec49eb3"
format_stamp: "Formatted at 2026-09-25 02:46:55 on dist-test-slave-n563"
server_key: "d6b10301246533dd1471b375a470c30c"
server_key_iv: "7087cef70c99b9eb950fb82d6f978745"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:55.517422 18089 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:46:55.532905 18113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:55.534550 18089 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:46:55.534739 18089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "eb42d44fb6de4b34a056e8762ec49eb3"
format_stamp: "Formatted at 2026-09-25 02:46:55 on dist-test-slave-n563"
server_key: "d6b10301246533dd1471b375a470c30c"
server_key_iv: "7087cef70c99b9eb950fb82d6f978745"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:55.534981 18089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:46:55.552618 18089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:55.557194 18089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:55.557520 18089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:55.558583 18089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:46:55.560017 18089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:46:55.560144 18089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:55.560290 18089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:46:55.560365 18089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:55.610075 18089 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:43305
I20260925 02:46:55.610105 18229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:43305 every 8 connection(s)
I20260925 02:46:55.611980 18089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:46:55.619093 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 18089
I20260925 02:46:55.619313 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:46:55.619813 5346 external_mini_cluster.cc:1428] Setting key fc9b292b0e4f19f73e5b995f8e5ae926
Sep 25 02:46:55 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304415, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:46:55.640534 18232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:55.614906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:43325 (local address 127.5.56.190:42991)
0925 02:46:55.615453 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:55.615460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:55.616604 (+ 1144us) server_negotiation.cc:407] Connection header received
0925 02:46:55.619755 (+ 3151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:55.619762 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:55.619879 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:55.619995 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:55.622216 (+ 2221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:55.623435 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:55.626170 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:55.626980 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:55.627132 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:55.632044 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:55.632053 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:55.632061 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:55.632107 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:55.636065 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:55.637097 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:55.637106 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:55.637113 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:55.637273 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:55.637825 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:55.637833 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:55.637839 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:55.638258 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:55.638508 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:55.639591 (+ 1083us) server_negotiation.cc:299] Negotiation successful
0925 02:46:55.639985 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":287,"threads_started":1}
I20260925 02:46:55.641660 18233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:55.615467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42991 (local address 127.5.56.130:43325)
0925 02:46:55.616390 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:55.616455 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:55.619363 (+ 2908us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:55.620324 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:55.620338 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:55.621015 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:46:55.622008 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:55.622039 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:55.623663 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:55.623672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:55.625977 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:55.625998 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:55.627259 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:55.627269 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:55.627627 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:55.628502 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:46:55.628539 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:46:55.631770 (+ 3231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:46:55.636252 (+ 4482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:55.636263 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:55.636285 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:55.636957 (+ 672us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:55.637409 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:55.637414 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:55.637419 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:55.637696 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:55.638494 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:46:55.638505 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:46:55.639018 (+ 513us) client_negotiation.cc:769] Sending connection context
0925 02:46:55.639421 (+ 403us) client_negotiation.cc:240] Negotiation successful
0925 02:46:55.640063 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":393,"threads_started":1}
I20260925 02:46:55.643545 18230 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42991
I20260925 02:46:55.643932 18230 heartbeater.cc:464] Registering TS with master...
I20260925 02:46:55.644917 18230 heartbeater.cc:511] Master 127.5.56.190:42991 requested a full tablet report, sending...
I20260925 02:46:55.646860 17867 ts_manager.cc:194] Registered new tserver with Master: eb42d44fb6de4b34a056e8762ec49eb3 (127.5.56.130:43305)
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
I20260925 02:46:55.647967 17867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:43325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:46:55.678913 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:48043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42991
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:46:55.972841 18083 heartbeater.cc:503] Master 127.5.56.190:42991 was elected leader, sending a full tablet report...
W20260925 02:46:56.024296 18236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:56.024642 18236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:56.024783 18236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:56.032146 18236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:56.032281 18236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:56.032459 18236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:46:56.042742 18236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.5.56.148:48043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:56.044140 18236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:56.045485 18236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:56.059634 18244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:56.059692 18242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:56.059697 18241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:56.060773 18236 server_base.cc:1034] running on GCE node
I20260925 02:46:56.061474 18236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:56.062608 18236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:56.063824 18236 hybrid_clock.cc:648] HybridClock initialized: now 1790304416063752 us; error 62 us; skew 500 ppm
I20260925 02:46:56.064150 18236 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:56 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304416, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:56.067720 18236 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:46:56.069311 18236 webserver.cc:460] Webserver started at http://127.5.56.131:33641/ using document root <none> and password file <none>
I20260925 02:46:56.070102 18236 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:56.070266 18236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:46:56.070667 18236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 02:46:56 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304416, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:56.089118 18236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "9793c0aa22c74e8ca16d7f60038443f5"
format_stamp: "Formatted at 2026-09-25 02:46:56 on dist-test-slave-n563"
server_key: "3a059307905c3d14a2f4c793c4b0f5b7"
server_key_iv: "014b16962365cedb7ef7f3c1ece46bdb"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:56.090284 18236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "9793c0aa22c74e8ca16d7f60038443f5"
format_stamp: "Formatted at 2026-09-25 02:46:56 on dist-test-slave-n563"
server_key: "3a059307905c3d14a2f4c793c4b0f5b7"
server_key_iv: "014b16962365cedb7ef7f3c1ece46bdb"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:56.097586 18236 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260925 02:46:56.112776 18251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:56.114362 18236 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:46:56.114645 18236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "9793c0aa22c74e8ca16d7f60038443f5"
format_stamp: "Formatted at 2026-09-25 02:46:56 on dist-test-slave-n563"
server_key: "3a059307905c3d14a2f4c793c4b0f5b7"
server_key_iv: "014b16962365cedb7ef7f3c1ece46bdb"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:56.114894 18236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:46:56.139189 18236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:56.143438 18236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:56.143769 18236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:56.144811 18236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:46:56.146193 18236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:46:56.146265 18236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:56.146452 18236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:46:56.146510 18236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 02:46:56.179859 18236 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:34019
I20260925 02:46:56.179980 18364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:34019 every 8 connection(s)
I20260925 02:46:56.181268 18236 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:46:56.185245 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 18236
I20260925 02:46:56.185449 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:46:56.185989 5346 external_mini_cluster.cc:1428] Setting key 102fb92dba76173e88deedb9ee9adf9d
Sep 25 02:46:56 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304416, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:46:56.201922 18367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:56.184205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:54407 (local address 127.5.56.190:42991)
0925 02:46:56.184770 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:56.184776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:56.185977 (+ 1201us) server_negotiation.cc:407] Connection header received
0925 02:46:56.188204 (+ 2227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:56.188209 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:56.188336 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:56.188443 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:56.189946 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:56.190752 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:56.192229 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:56.192663 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:56.192755 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:56.196003 (+ 3248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:56.196007 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:56.196010 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:56.196044 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:56.198564 (+ 2520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:56.199249 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:56.199253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:56.199255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:56.199399 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:56.199869 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:56.199877 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:56.199880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:56.200234 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:56.200381 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:56.201305 (+ 924us) server_negotiation.cc:299] Negotiation successful
0925 02:46:56.201550 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":253,"threads_started":1}
I20260925 02:46:56.203012 18368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:56.184891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42991 (local address 127.5.56.131:54407)
0925 02:46:56.185803 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:56.185848 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:56.187892 (+ 2044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:56.188649 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:56.188657 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:56.189193 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:46:56.189798 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:56.189815 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:56.190877 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:56.190880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:56.192111 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:56.192122 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:56.192850 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:56.192855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:56.193047 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:56.193712 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:46:56.193740 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:46:56.195794 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:46:56.198696 (+ 2902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:56.198705 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:56.198719 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:56.199154 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:56.199510 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:56.199513 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:56.199515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:56.199715 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:56.200468 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:46:56.200480 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:46:56.200882 (+ 402us) client_negotiation.cc:769] Sending connection context
0925 02:46:56.201206 (+ 324us) client_negotiation.cc:240] Negotiation successful
0925 02:46:56.201713 (+ 507us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":401,"threads_started":1}
I20260925 02:46:56.204311 18365 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42991
I20260925 02:46:56.204617 18365 heartbeater.cc:464] Registering TS with master...
I20260925 02:46:56.205339 18365 heartbeater.cc:511] Master 127.5.56.190:42991 requested a full tablet report, sending...
I20260925 02:46:56.206779 17867 ts_manager.cc:194] Registered new tserver with Master: 9793c0aa22c74e8ca16d7f60038443f5 (127.5.56.131:34019)
I20260925 02:46:56.207571 17867 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:54407
I20260925 02:46:56.213968 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:46:56.215667 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 17948
Sep 25 02:46:56 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:46:56.232263 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:56.216598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46128 (local address 127.5.56.129:46811)
0925 02:46:56.217248 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:56.217255 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:56.217280 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:46:56.217344 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:56.217352 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:56.217622 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:56.217753 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:56.219296 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:56.220383 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:56.221796 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:56.222353 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:56.222515 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:56.226824 (+ 4309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:56.226837 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:56.226854 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:56.226894 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:56.229579 (+ 2685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:56.230294 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:56.230301 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:56.230303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:56.230530 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:56.230867 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:56.230870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:56.230872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:56.231195 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:56.231436 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:56.231662 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 02:46:56.231861 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":335,"threads_started":1}
I20260925 02:46:56.233655 18078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:46:56.318480 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 17948
I20260925 02:46:56.336222 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 18089
Sep 25 02:46:56 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:46:56.352494 18372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:56.337148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39584 (local address 127.5.56.130:43305)
0925 02:46:56.337705 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:56.337713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:56.337739 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:46:56.337800 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:56.337807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:56.338029 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:56.338203 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:56.339563 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:56.340564 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:56.341848 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:56.342607 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:56.342750 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:56.347092 (+ 4342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:56.347103 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:56.347119 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:56.347160 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:56.349828 (+ 2668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:56.350573 (+ 745us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:56.350579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:56.350582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:56.350739 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:56.351073 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:56.351076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:56.351078 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:56.351408 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:56.351623 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:56.351890 (+ 267us) server_negotiation.cc:299] Negotiation successful
0925 02:46:56.352091 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":287,"threads_started":1}
I20260925 02:46:56.353782 18225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:46:56.438601 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 18089
I20260925 02:46:56.459441 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 18236
Sep 25 02:46:56 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304412, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:46:56.475723 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:56.460481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60786 (local address 127.5.56.131:34019)
0925 02:46:56.461122 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:56.461130 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:56.461159 (+ 29us) server_negotiation.cc:407] Connection header received
0925 02:46:56.461219 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:56.461225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:56.461456 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:56.461583 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:56.462946 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:56.463936 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:56.465142 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:56.465702 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:56.465798 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:56.470266 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:56.470279 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:56.470296 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:56.470333 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:56.473000 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:56.473726 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:56.473734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:56.473736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:56.473882 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:56.474202 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:56.474205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:56.474207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:56.474621 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:56.474878 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:56.475117 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 02:46:56.475333 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":341,"threads_started":1}
I20260925 02:46:56.477051 18360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:46:56.561513 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 18236
I20260925 02:46:56.578027 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 17835
I20260925 02:46:56.579478 17898 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:46:56.690569 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 17835
I20260925 02:46:56.713307 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:42991
--webserver_interface=127.5.56.190
--webserver_port=35021
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:42991
--ranger_config_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:48043/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.56.148:48043/kms with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:46:57.048070 18377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:57.048448 18377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:57.048602 18377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:57.056077 18377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:46:57.056231 18377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:57.056342 18377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:57.056437 18377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:46:57.056522 18377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:46:57.067178 18377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.56.148:48043/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:42991
--ranger_config_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:42991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=35021
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:57.068598 18377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:57.070133 18377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:57.082362 18382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:57.082373 18383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:57.082370 18385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:57.083156 18377 server_base.cc:1034] running on GCE node
I20260925 02:46:57.083993 18377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:57.085533 18377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:57.086797 18377 hybrid_clock.cc:648] HybridClock initialized: now 1790304417086733 us; error 48 us; skew 500 ppm
I20260925 02:46:57.087186 18377 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:57 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304417, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:57.090205 18377 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:46:57.091691 18377 webserver.cc:460] Webserver started at http://127.5.56.190:35021/ using document root <none> and password file <none>
I20260925 02:46:57.092372 18377 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:57.092507 18377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:46:57 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304417, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:57.145183 18377 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:46:57.148510 18394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:57.149868 18377 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:46:57.150070 18377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "605bc8465e79412399a52e2fbd678dcd"
format_stamp: "Formatted at 2026-09-25 02:46:53 on dist-test-slave-n563"
server_key: "0203859277a20bd98df62840ff2cb741"
server_key_iv: "f46954b56c4355b923aa6dd9156a46b4"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:57.151166 18377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:46:57.164635 18377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:57.168547 18377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:57.168859 18377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:57.187837 18377 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:42991
I20260925 02:46:57.187878 18446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:42991 every 8 connection(s)
I20260925 02:46:57.189409 18377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:46:57.196446 18447 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:46:57.198423 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 18377
I20260925 02:46:57.199282 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:48043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:46811
--local_ip_for_outbound_sockets=127.5.56.129
--tserver_master_addrs=127.5.56.190:42991
--webserver_port=45843
--webserver_interface=127.5.56.129
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.56.148:48043/kms with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:46:57.199911 18447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd: Bootstrap starting.
I20260925 02:46:57.218927 18447 log.cc:824] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd: Log is configured to *not* fsync() on all Append() calls
I20260925 02:46:57.228478 18447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:46:57.229144 18447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd: Bootstrap complete.
I20260925 02:46:57.243395 18447 raft_consensus.cc:348] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } }
I20260925 02:46:57.244524 18447 raft_consensus.cc:729] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 605bc8465e79412399a52e2fbd678dcd, State: Initialized, Role: FOLLOWER
I20260925 02:46:57.245255 18447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } }
I20260925 02:46:57.245503 18447 raft_consensus.cc:388] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:46:57.245633 18447 raft_consensus.cc:482] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:46:57.245826 18447 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:46:57.249794 18447 raft_consensus.cc:504] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } }
I20260925 02:46:57.250326 18447 leader_election.cc:302] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 605bc8465e79412399a52e2fbd678dcd; no voters:
I20260925 02:46:57.251072 18447 leader_election.cc:288] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:46:57.251230 18451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:46:57.252453 18451 raft_consensus.cc:686] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [term 2 LEADER]: Becoming Leader. State: Replica: 605bc8465e79412399a52e2fbd678dcd, State: Running, Role: LEADER
I20260925 02:46:57.253122 18451 consensus_queue.cc:237] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } }
I20260925 02:46:57.253438 18447 sys_catalog.cc:564] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:46:57.255772 18453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 605bc8465e79412399a52e2fbd678dcd. Latest consensus state: current_term: 2 leader_uuid: "605bc8465e79412399a52e2fbd678dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } } }
I20260925 02:46:57.256223 18453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [sys.catalog]: This master's current role is: LEADER
I20260925 02:46:57.256870 18452 sys_catalog.cc:455] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "605bc8465e79412399a52e2fbd678dcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "605bc8465e79412399a52e2fbd678dcd" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42991 } } }
I20260925 02:46:57.257202 18452 sys_catalog.cc:458] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd [sys.catalog]: This master's current role is: LEADER
I20260925 02:46:57.265113 18457 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:46:57.272382 18457 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:46:57.273888 18457 catalog_manager.cc:1259] Loaded cluster ID: 0fd90f3ea38445239483995cda65d0c3
I20260925 02:46:57.274066 18457 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:46:57.278180 18457 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:46:57.280148 18457 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd: Loaded TSK: 0
I20260925 02:46:57.281797 18457 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:46:57.288220 18447 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260925 02:46:57.660275 18449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:57.660838 18449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:57.661053 18449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:57.673465 18449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:57.673709 18449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:57.673988 18449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:46:57.690286 18449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.56.148:48043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:46811
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=45843
--tserver_master_addrs=127.5.56.190:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:57.692396 18449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:57.693885 18449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:57.715466 18486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:57.722788 18487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:57.725090 18489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:57.725879 18449 server_base.cc:1034] running on GCE node
I20260925 02:46:57.727025 18449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:57.728425 18449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:57.729743 18449 hybrid_clock.cc:648] HybridClock initialized: now 1790304417729649 us; error 84 us; skew 500 ppm
I20260925 02:46:57.730204 18449 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:57 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304417, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:57.734753 18449 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:46:57.740271 18449 webserver.cc:460] Webserver started at http://127.5.56.129:45843/ using document root <none> and password file <none>
I20260925 02:46:57.741346 18449 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:57.741505 18449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:46:57 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304417, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:57.802435 18449 fs_manager.cc:709] Time spent opening directory manager: real 0.039s user 0.008s sys 0.028s
I20260925 02:46:57.812871 18497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:57.814947 18449 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.009s
I20260925 02:46:57.815153 18449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "97c83d8b2dde4fa4a844a00e6def068a"
format_stamp: "Formatted at 2026-09-25 02:46:54 on dist-test-slave-n563"
server_key: "4b799560b63925317806a34e3248ef59"
server_key_iv: "3ce038a59fb405b70394ab12862f4c8f"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:57.816702 18449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:46:57.838109 18449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:57.843755 18449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:57.844097 18449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:57.845456 18449 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:46:57.847368 18449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:46:57.847491 18449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:57.847610 18449 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:46:57.847702 18449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:57.922216 18449 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:46811
I20260925 02:46:57.922322 18613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:46811 every 8 connection(s)
I20260925 02:46:57.924255 18449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:46:57.929574 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 18449
I20260925 02:46:57.930424 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:48043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:43305
--local_ip_for_outbound_sockets=127.5.56.130
--tserver_master_addrs=127.5.56.190:42991
--webserver_port=42523
--webserver_interface=127.5.56.130
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.56.148:48043/kms with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:46:57 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304417, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:46:57.969808 18616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:57.927639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42991 (local address 127.5.56.129:36101)
0925 02:46:57.928483 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:57.928535 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:57.931159 (+ 2624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:57.933616 (+ 2457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:57.933627 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:57.934264 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:46:57.935043 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:57.935066 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:57.937513 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:57.937519 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:57.939625 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:57.939640 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:57.951125 (+ 11485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:57.951135 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:57.951584 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:57.952508 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:46:57.952556 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:46:57.959073 (+ 6517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:46:57.964069 (+ 4996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:57.964080 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:57.964098 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:57.964815 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:57.965473 (+ 658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:57.965478 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:57.965483 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:57.965779 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:57.966901 (+ 1122us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:46:57.966911 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:46:57.967371 (+ 460us) client_negotiation.cc:769] Sending connection context
0925 02:46:57.967677 (+ 306us) client_negotiation.cc:240] Negotiation successful
0925 02:46:57.968295 (+ 618us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":334,"threads_started":1}
I20260925 02:46:57.972545 18618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:57.927577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:36101 (local address 127.5.56.190:42991)
0925 02:46:57.932025 (+ 4448us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:57.932047 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:57.932100 (+ 53us) server_negotiation.cc:407] Connection header received
0925 02:46:57.932529 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:57.932551 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:57.933039 (+ 488us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:57.933456 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:57.935296 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:57.937305 (+ 2009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:57.939811 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:57.950708 (+ 10897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:57.951074 (+ 366us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:57.959573 (+ 8499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:57.959589 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:57.959616 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:57.959684 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:57.963832 (+ 4148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:57.965066 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:57.965077 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:57.965087 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:57.965308 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:57.965929 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:57.965934 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:57.965938 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:57.966716 (+ 778us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:57.970360 (+ 3644us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:57.970718 (+ 358us) server_negotiation.cc:299] Negotiation successful
0925 02:46:57.971192 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4130,"thread_start_us":3845,"threads_started":1}
I20260925 02:46:57.976109 18614 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42991
I20260925 02:46:57.976445 18614 heartbeater.cc:464] Registering TS with master...
I20260925 02:46:57.977327 18614 heartbeater.cc:511] Master 127.5.56.190:42991 requested a full tablet report, sending...
W20260925 02:46:57.980865 18614 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:42991 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:46:57.981097 18614 heartbeater.cc:464] Registering TS with master...
I20260925 02:46:57.981411 18614 heartbeater.cc:511] Master 127.5.56.190:42991 requested a full tablet report, sending...
I20260925 02:46:57.982852 18614 heartbeater.cc:464] Registering TS with master...
I20260925 02:46:57.983186 18614 heartbeater.cc:511] Master 127.5.56.190:42991 requested a full tablet report, sending...
W20260925 02:46:57.984292 18614 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:46:58.367715 18619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:58.368157 18619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:58.368355 18619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:58.378259 18619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:58.378471 18619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:58.378700 18619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:46:58.393931 18619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.56.148:48043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:43305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=42523
--tserver_master_addrs=127.5.56.190:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:58.395457 18619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:58.397008 18619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:58.436309 18633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:58.437661 18634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:58.454468 18638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:58.454866 18619 server_base.cc:1034] running on GCE node
I20260925 02:46:58.455873 18619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:58.465327 18619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:58.466711 18619 hybrid_clock.cc:648] HybridClock initialized: now 1790304418466624 us; error 87 us; skew 500 ppm
I20260925 02:46:58.467198 18619 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:58 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304418, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:58.477252 18619 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:46:58.479918 18619 webserver.cc:460] Webserver started at http://127.5.56.130:42523/ using document root <none> and password file <none>
I20260925 02:46:58.480974 18619 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:58.481140 18619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:46:58 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304418, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:58.507520 18619 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260925 02:46:58.514541 18646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:58.516686 18619 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.002s
I20260925 02:46:58.516907 18619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "eb42d44fb6de4b34a056e8762ec49eb3"
format_stamp: "Formatted at 2026-09-25 02:46:55 on dist-test-slave-n563"
server_key: "d6b10301246533dd1471b375a470c30c"
server_key_iv: "7087cef70c99b9eb950fb82d6f978745"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:58.518450 18619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 25 02:46:58 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790304418, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:58.539456 18619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:58.545186 18619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:58.545622 18619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:58.547276 18619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:46:58.549135 18619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:46:58.549307 18619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:58.549471 18619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:46:58.549564 18619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:58.599545 18619 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:43305
I20260925 02:46:58.599645 18760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:43305 every 8 connection(s)
I20260925 02:46:58.601536 18619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:46:58.604760 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 18619
I20260925 02:46:58.605575 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.5.56.148:48043/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:34019
--local_ip_for_outbound_sockets=127.5.56.131
--tserver_master_addrs=127.5.56.190:42991
--webserver_port=33641
--webserver_interface=127.5.56.131
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43033
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.5.56.148:48043/kms with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:46:58 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304418, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:46:58.656272 18765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:58.608626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:36955 (local address 127.5.56.190:42991)
0925 02:46:58.628970 (+ 20344us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:58.628980 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:58.635665 (+ 6685us) server_negotiation.cc:407] Connection header received
0925 02:46:58.635765 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:58.635770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:58.635868 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:58.635984 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:58.638439 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:58.639643 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:58.642227 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:58.642952 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:58.643077 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:58.647823 (+ 4746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:58.647831 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:58.647835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:58.647880 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:58.651685 (+ 3805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:58.652779 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:58.652786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:58.652790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:58.652952 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:58.653537 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:58.653542 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:58.653546 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:58.654026 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:58.654219 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:58.655533 (+ 1314us) server_negotiation.cc:299] Negotiation successful
0925 02:46:58.655790 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20131,"thread_start_us":20342,"threads_started":1}
I20260925 02:46:58.658983 18764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:58.605374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42991 (local address 127.5.56.130:36955)
0925 02:46:58.632461 (+ 27087us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:58.632533 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:58.635194 (+ 2661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:58.636503 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:58.636517 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:58.637249 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:46:58.638173 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:58.638201 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:58.639895 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:58.639901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:58.642015 (+ 2114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:58.642041 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:58.643484 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:58.643492 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:58.643853 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:58.644705 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:46:58.644738 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:46:58.647494 (+ 2756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:46:58.651907 (+ 4413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:58.651919 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:58.651941 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:58.652607 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:58.653121 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:58.653127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:58.653131 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:58.653406 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:58.654334 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:46:58.654344 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:46:58.655005 (+ 661us) client_negotiation.cc:769] Sending connection context
0925 02:46:58.656839 (+ 1834us) client_negotiation.cc:240] Negotiation successful
0925 02:46:58.657438 (+ 599us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24050,"thread_start_us":23777,"threads_started":1}
I20260925 02:46:58.661104 18762 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42991
I20260925 02:46:58.661455 18762 heartbeater.cc:464] Registering TS with master...
I20260925 02:46:58.662415 18762 heartbeater.cc:511] Master 127.5.56.190:42991 requested a full tablet report, sending...
I20260925 02:46:58.664714 18411 ts_manager.cc:194] Registered new tserver with Master: eb42d44fb6de4b34a056e8762ec49eb3 (127.5.56.130:43305)
I20260925 02:46:58.666932 18411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:36955
I20260925 02:46:58.984447 18614 heartbeater.cc:464] Registering TS with master...
I20260925 02:46:58.984874 18614 heartbeater.cc:511] Master 127.5.56.190:42991 requested a full tablet report, sending...
I20260925 02:46:58.986138 18411 ts_manager.cc:194] Registered new tserver with Master: 97c83d8b2dde4fa4a844a00e6def068a (127.5.56.129:46811)
I20260925 02:46:58.987375 18411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:36101
Sep 25 02:46:58 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304418, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
W20260925 02:46:59.127980 18766 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:46:59.128512 18766 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:46:59.128741 18766 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:46:59.139710 18766 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:46:59.139954 18766 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:46:59.140220 18766 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:46:59.155639 18766 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43033
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.5.56.148:48043/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:34019
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=33641
--tserver_master_addrs=127.5.56.190:42991
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:46:59.157228 18766 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:46:59.158686 18766 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:46:59.177980 18771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:46:59.179980 18772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:59.200575 18766 server_base.cc:1034] running on GCE node
W20260925 02:46:59.201028 18774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:46:59.201917 18766 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:46:59.205175 18766 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:46:59.206504 18766 hybrid_clock.cc:648] HybridClock initialized: now 1790304419206415 us; error 71 us; skew 500 ppm
I20260925 02:46:59.206871 18766 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:46:59 dist-test-slave-n563 krb5kdc[17384](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304419, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:46:59.210320 18766 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:46:59.212231 18766 webserver.cc:460] Webserver started at http://127.5.56.131:33641/ using document root <none> and password file <none>
I20260925 02:46:59.212975 18766 fs_manager.cc:359] Metadata directory not provided
I20260925 02:46:59.213133 18766 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 02:46:59 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304419, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
I20260925 02:46:59.244235 18766 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.003s sys 0.003s
I20260925 02:46:59.247432 18782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:59.249022 18766 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:46:59.249212 18766 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "9793c0aa22c74e8ca16d7f60038443f5"
format_stamp: "Formatted at 2026-09-25 02:46:56 on dist-test-slave-n563"
server_key: "3a059307905c3d14a2f4c793c4b0f5b7"
server_key_iv: "014b16962365cedb7ef7f3c1ece46bdb"
server_key_version: "kuduclusterkey@0"
I20260925 02:46:59.250281 18766 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:46:59.276517 18766 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:46:59.281004 18766 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:46:59.281388 18766 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:46:59.282522 18766 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:46:59.284031 18766 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:46:59.284147 18766 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:59.284273 18766 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:46:59.284355 18766 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:46:59.357856 18766 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:34019
I20260925 02:46:59.357906 18894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:34019 every 8 connection(s)
I20260925 02:46:59.359925 18766 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:46:59.369818 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 18766
Sep 25 02:46:59 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304418, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for HTTP/127.5.56.148@KRBTEST.COM
Sep 25 02:46:59 dist-test-slave-n563 krb5kdc[17384](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304419, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:46:59.423267 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.370987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.5.56.190:42991)
0925 02:46:59.393932 (+ 22945us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:59.393946 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:59.394850 (+ 904us) server_negotiation.cc:407] Connection header received
0925 02:46:59.394914 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:59.394920 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:59.395008 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:59.395108 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:59.399779 (+ 4671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.400932 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.407604 (+ 6672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.408268 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.408607 (+ 339us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.414155 (+ 5548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:59.414162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:59.414167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:59.414232 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:59.417904 (+ 3672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:59.419030 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:59.419036 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:59.419040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:59.419186 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:59.420164 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:59.420170 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:59.420174 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:59.421486 (+ 1312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:59.421780 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:59.422525 (+ 745us) server_negotiation.cc:299] Negotiation successful
0925 02:46:59.422848 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22764,"thread_start_us":19178,"threads_started":1}
I20260925 02:46:59.430631 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.365987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:60757 (local address 127.5.56.190:42991)
0925 02:46:59.390524 (+ 24537us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:59.390532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:59.390559 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:46:59.393103 (+ 2544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:59.393110 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:59.393220 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:59.393350 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:59.403478 (+ 10128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.404606 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.410251 (+ 5645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.410954 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.411054 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.419456 (+ 8402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:59.419463 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:59.419468 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:59.419505 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:59.425141 (+ 5636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:59.426262 (+ 1121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:59.426267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:59.426270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:59.426447 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:59.427041 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:59.427047 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:59.427050 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:59.427924 (+ 874us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:59.428072 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:59.429758 (+ 1686us) server_negotiation.cc:299] Negotiation successful
0925 02:46:59.430036 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24347,"thread_start_us":27612,"threads_started":1}
I20260925 02:46:59.432216 18900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.368955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:42991 (local address 127.5.56.131:60757)
0925 02:46:59.382271 (+ 13316us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:59.382352 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:59.392565 (+ 10213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:59.395682 (+ 3117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:59.395698 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:59.396455 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:46:59.397422 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.397453 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.405122 (+ 7669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.405128 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.410050 (+ 4922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.410083 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.411473 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.411482 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.411865 (+ 383us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.412740 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:46:59.412775 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:46:59.418815 (+ 6040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:46:59.425376 (+ 6561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:59.425390 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:59.425414 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:59.426095 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:59.426601 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:46:59.426607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:46:59.426611 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:46:59.426902 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:46:59.428162 (+ 1260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:46:59.428171 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:46:59.429077 (+ 906us) client_negotiation.cc:769] Sending connection context
0925 02:46:59.429620 (+ 543us) client_negotiation.cc:240] Negotiation successful
0925 02:46:59.430453 (+ 833us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12872,"thread_start_us":13444,"threads_started":1}
I20260925 02:46:59.434039 18896 heartbeater.cc:347] Connected to a master server at 127.5.56.190:42991
I20260925 02:46:59.434370 18896 heartbeater.cc:464] Registering TS with master...
I20260925 02:46:59.435369 18896 heartbeater.cc:511] Master 127.5.56.190:42991 requested a full tablet report, sending...
I20260925 02:46:59.437274 18411 ts_manager.cc:194] Registered new tserver with Master: 9793c0aa22c74e8ca16d7f60038443f5 (127.5.56.131:34019)
I20260925 02:46:59.438408 18411 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:60757
I20260925 02:46:59.442255 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:46:59.484709 18901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.458730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.5.56.190:42991)
0925 02:46:59.458939 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:59.458944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:59.466054 (+ 7110us) server_negotiation.cc:407] Connection header received
0925 02:46:59.466663 (+ 609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:59.466670 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:59.466791 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:59.466878 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:59.469250 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.470367 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.472537 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.473216 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.473341 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.477290 (+ 3949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:59.477298 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:59.477303 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:59.477353 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:59.481314 (+ 3961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:59.482366 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:59.482372 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:59.482376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:59.482586 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:59.483158 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:59.483164 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:59.483167 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:59.483479 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:59.483732 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:59.484100 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 02:46:59.484341 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 02:46:59.490142 18411 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34148:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:46:59.500845 18411 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:46:59.577868 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.537963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40070 (local address 127.5.56.131:34019)
0925 02:46:59.544380 (+ 6417us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:59.544394 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:59.559479 (+ 15085us) server_negotiation.cc:407] Connection header received
0925 02:46:59.559892 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:59.559903 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:59.560240 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:59.560460 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:46:59.562255 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.564054 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.571129 (+ 7075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.572780 (+ 1651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.572888 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.573136 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:59.577264 (+ 4128us) server_negotiation.cc:299] Negotiation successful
0925 02:46:59.577467 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6186,"thread_start_us":6000,"threads_started":1}
I20260925 02:46:59.579124 18912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.537251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:34019 (local address 127.0.0.1:40070)
0925 02:46:59.559341 (+ 22090us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:59.559365 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:59.559609 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:59.560941 (+ 1332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:59.560949 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:59.560983 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:46:59.561371 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.561385 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.565633 (+ 4248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.565639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.568147 (+ 2508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.568164 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.575186 (+ 7022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.575194 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.575471 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.575543 (+ 72us) client_negotiation.cc:769] Sending connection context
0925 02:46:59.578495 (+ 2952us) client_negotiation.cc:240] Negotiation successful
0925 02:46:59.578744 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19981,"thread_start_us":5374,"threads_started":1}
I20260925 02:46:59.584044 18914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.533857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:43305 (local address 127.0.0.1:54408)
0925 02:46:59.543213 (+ 9356us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:59.543272 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:59.543537 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:59.576907 (+ 33370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:59.576911 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:59.576934 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:46:59.577307 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.577317 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.579208 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.579216 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.581276 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.581290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.583264 (+ 1974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.583271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.583454 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.583490 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:46:59.583563 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 02:46:59.583710 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8957,"thread_start_us":24729,"threads_started":1}
I20260925 02:46:59.584004 18830 tablet_service.cc:1467] Processing CreateTablet for tablet 4358076c28254040a8e6ebb4ab9db147 (DEFAULT_TABLE table=test-table [id=9d013f5098424752801687f847bcf6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:46:59.585409 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.534068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.5.56.130:43305)
0925 02:46:59.576276 (+ 42208us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:59.576285 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:59.576329 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:46:59.576441 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:59.576450 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:59.576696 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:59.576932 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:46:59.577481 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.579025 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.581508 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.582980 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.584627 (+ 1647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.584833 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:59.584918 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 02:46:59.585084 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41999,"thread_start_us":41900,"threads_started":1}
I20260925 02:46:59.585914 18830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4358076c28254040a8e6ebb4ab9db147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:46:59.588949 18694 tablet_service.cc:1467] Processing CreateTablet for tablet 4358076c28254040a8e6ebb4ab9db147 (DEFAULT_TABLE table=test-table [id=9d013f5098424752801687f847bcf6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:46:59.590179 18694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4358076c28254040a8e6ebb4ab9db147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:46:59.598647 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.559032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46811 (local address 127.0.0.1:41578)
0925 02:46:59.575564 (+ 16532us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:59.575593 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:59.575784 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:59.578437 (+ 2653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:59.578443 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:59.578471 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:46:59.578827 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.578837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.591583 (+ 12746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.591589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.595355 (+ 3766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.595375 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.597738 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.597747 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.597938 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.597975 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:46:59.598072 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 02:46:59.598249 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16323,"thread_start_us":16652,"threads_started":1}
I20260925 02:46:59.599545 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.559431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.5.56.129:46811)
0925 02:46:59.576351 (+ 16920us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:59.576357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:59.576382 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:46:59.576459 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:59.576464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:59.576682 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:59.578986 (+ 2304us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:46:59.579263 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.591362 (+ 12099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.595608 (+ 4246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.597152 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.597320 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.597573 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:59.599027 (+ 1454us) server_negotiation.cc:299] Negotiation successful
0925 02:46:59.599207 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16744,"thread_start_us":18009,"threads_started":1}
I20260925 02:46:59.606594 18547 tablet_service.cc:1467] Processing CreateTablet for tablet 4358076c28254040a8e6ebb4ab9db147 (DEFAULT_TABLE table=test-table [id=9d013f5098424752801687f847bcf6f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:46:59.608639 18547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4358076c28254040a8e6ebb4ab9db147. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:46:59.610566 18918 tablet_bootstrap.cc:492] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5: Bootstrap starting.
I20260925 02:46:59.614516 18918 tablet_bootstrap.cc:654] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5: Neither blocks nor log segments found. Creating new log.
I20260925 02:46:59.615979 18918 log.cc:824] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:46:59.618080 18919 tablet_bootstrap.cc:492] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3: Bootstrap starting.
I20260925 02:46:59.627068 18918 tablet_bootstrap.cc:492] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5: No bootstrap required, opened a new log
I20260925 02:46:59.627589 18918 ts_tablet_manager.cc:1397] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.006s
I20260925 02:46:59.633463 18921 tablet_bootstrap.cc:492] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a: Bootstrap starting.
I20260925 02:46:59.634917 18919 tablet_bootstrap.cc:654] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3: Neither blocks nor log segments found. Creating new log.
I20260925 02:46:59.636471 18919 log.cc:824] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:46:59.639191 18921 tablet_bootstrap.cc:654] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a: Neither blocks nor log segments found. Creating new log.
I20260925 02:46:59.640542 18921 log.cc:824] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:46:59.647838 18918 raft_consensus.cc:348] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43305 } } peers { permanent_uuid: "9793c0aa22c74e8ca16d7f60038443f5" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34019 } } peers { permanent_uuid: "97c83d8b2dde4fa4a844a00e6def068a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46811 } }
I20260925 02:46:59.650192 18921 tablet_bootstrap.cc:492] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a: No bootstrap required, opened a new log
I20260925 02:46:59.650513 18919 tablet_bootstrap.cc:492] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3: No bootstrap required, opened a new log
I20260925 02:46:59.650736 18921 ts_tablet_manager.cc:1397] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260925 02:46:59.650878 18919 ts_tablet_manager.cc:1397] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.000s
I20260925 02:46:59.654357 18918 raft_consensus.cc:374] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:46:59.654948 18918 raft_consensus.cc:729] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9793c0aa22c74e8ca16d7f60038443f5, State: Initialized, Role: FOLLOWER
I20260925 02:46:59.662151 18921 raft_consensus.cc:348] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43305 } } peers { permanent_uuid: "9793c0aa22c74e8ca16d7f60038443f5" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34019 } } peers { permanent_uuid: "97c83d8b2dde4fa4a844a00e6def068a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46811 } }
I20260925 02:46:59.662829 18921 raft_consensus.cc:374] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:46:59.662997 18921 raft_consensus.cc:729] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c83d8b2dde4fa4a844a00e6def068a, State: Initialized, Role: FOLLOWER
I20260925 02:46:59.662887 18919 raft_consensus.cc:348] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43305 } } peers { permanent_uuid: "9793c0aa22c74e8ca16d7f60038443f5" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34019 } } peers { permanent_uuid: "97c83d8b2dde4fa4a844a00e6def068a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46811 } }
I20260925 02:46:59.663760 18921 consensus_queue.cc:260] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43305 } } peers { permanent_uuid: "9793c0aa22c74e8ca16d7f60038443f5" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34019 } } peers { permanent_uuid: "97c83d8b2dde4fa4a844a00e6def068a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46811 } }
I20260925 02:46:59.663761 18918 consensus_queue.cc:260] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43305 } } peers { permanent_uuid: "9793c0aa22c74e8ca16d7f60038443f5" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34019 } } peers { permanent_uuid: "97c83d8b2dde4fa4a844a00e6def068a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46811 } }
I20260925 02:46:59.669528 18918 ts_tablet_manager.cc:1428] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5: Time spent starting tablet: real 0.042s user 0.012s sys 0.006s
I20260925 02:46:59.670528 18919 raft_consensus.cc:374] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:46:59.670912 18896 heartbeater.cc:503] Master 127.5.56.190:42991 was elected leader, sending a full tablet report...
I20260925 02:46:59.671933 18614 heartbeater.cc:503] Master 127.5.56.190:42991 was elected leader, sending a full tablet report...
I20260925 02:46:59.674649 18921 ts_tablet_manager.cc:1428] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a: Time spent starting tablet: real 0.024s user 0.020s sys 0.000s
I20260925 02:46:59.672490 18762 heartbeater.cc:503] Master 127.5.56.190:42991 was elected leader, sending a full tablet report...
I20260925 02:46:59.675927 18919 raft_consensus.cc:729] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb42d44fb6de4b34a056e8762ec49eb3, State: Initialized, Role: FOLLOWER
W20260925 02:46:59.677426 18615 tablet.cc:2369] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:46:59.677151 18919 consensus_queue.cc:260] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43305 } } peers { permanent_uuid: "9793c0aa22c74e8ca16d7f60038443f5" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34019 } } peers { permanent_uuid: "97c83d8b2dde4fa4a844a00e6def068a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46811 } }
I20260925 02:46:59.687763 18927 raft_consensus.cc:482] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:46:59.690062 18927 raft_consensus.cc:504] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43305 } } peers { permanent_uuid: "9793c0aa22c74e8ca16d7f60038443f5" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34019 } } peers { permanent_uuid: "97c83d8b2dde4fa4a844a00e6def068a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46811 } }
I20260925 02:46:59.689754 18919 ts_tablet_manager.cc:1428] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3: Time spent starting tablet: real 0.039s user 0.015s sys 0.004s
I20260925 02:46:59.695052 18927 leader_election.cc:288] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9793c0aa22c74e8ca16d7f60038443f5 (127.5.56.131:34019), 97c83d8b2dde4fa4a844a00e6def068a (127.5.56.129:46811)
I20260925 02:46:59.705518 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.693563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:51099 (local address 127.5.56.131:34019)
0925 02:46:59.693797 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:59.693804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:59.696695 (+ 2891us) server_negotiation.cc:407] Connection header received
0925 02:46:59.696913 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:59.696923 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:59.697030 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:59.697131 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:46:59.698119 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.699643 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.703089 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.704247 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.704391 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.704470 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:59.705143 (+ 673us) server_negotiation.cc:299] Negotiation successful
0925 02:46:59.705268 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:46:59.705574 18928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.692963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:34019 (local address 127.5.56.130:51099)
0925 02:46:59.696562 (+ 3599us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:59.696585 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:59.696759 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:59.697350 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:59.697358 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:59.697395 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:46:59.697895 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.697911 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.699873 (+ 1962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.699882 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.702913 (+ 3031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.702928 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.704820 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.704825 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.704955 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.704977 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:46:59.705089 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:46:59.705227 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3474,"thread_start_us":3327,"threads_started":1}
I20260925 02:46:59.706773 18850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4358076c28254040a8e6ebb4ab9db147" candidate_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793c0aa22c74e8ca16d7f60038443f5" is_pre_election: true
I20260925 02:46:59.707377 18850 raft_consensus.cc:2454] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb42d44fb6de4b34a056e8762ec49eb3 in term 0.
I20260925 02:46:59.708300 18649 leader_election.cc:302] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793c0aa22c74e8ca16d7f60038443f5, eb42d44fb6de4b34a056e8762ec49eb3; no voters:
I20260925 02:46:59.709245 18927 raft_consensus.cc:2781] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:46:59.709499 18927 raft_consensus.cc:482] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:46:59.709681 18927 raft_consensus.cc:3037] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:46:59.713032 18927 raft_consensus.cc:504] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43305 } } peers { permanent_uuid: "9793c0aa22c74e8ca16d7f60038443f5" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34019 } } peers { permanent_uuid: "97c83d8b2dde4fa4a844a00e6def068a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46811 } }
I20260925 02:46:59.714359 18850 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4358076c28254040a8e6ebb4ab9db147" candidate_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9793c0aa22c74e8ca16d7f60038443f5"
I20260925 02:46:59.714607 18850 raft_consensus.cc:3037] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:46:59.715730 18927 leader_election.cc:288] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [CANDIDATE]: Term 1 election: Requested vote from peers 9793c0aa22c74e8ca16d7f60038443f5 (127.5.56.131:34019), 97c83d8b2dde4fa4a844a00e6def068a (127.5.56.129:46811)
I20260925 02:46:59.717926 18850 raft_consensus.cc:2454] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb42d44fb6de4b34a056e8762ec49eb3 in term 1.
I20260925 02:46:59.718045 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.695636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:39607 (local address 127.5.56.129:46811)
0925 02:46:59.695876 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:59.695882 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:59.703112 (+ 7230us) server_negotiation.cc:407] Connection header received
0925 02:46:59.710771 (+ 7659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:59.710777 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:59.710876 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:59.710944 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:46:59.711861 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.712947 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.715519 (+ 2572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.716799 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.716968 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.717047 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:59.717609 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 02:46:59.717767 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:46:59.718641 18649 leader_election.cc:302] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9793c0aa22c74e8ca16d7f60038443f5, eb42d44fb6de4b34a056e8762ec49eb3; no voters:
I20260925 02:46:59.718961 18929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.695870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46811 (local address 127.5.56.130:39607)
0925 02:46:59.702963 (+ 7093us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:59.702986 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:46:59.710532 (+ 7546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:46:59.711155 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:46:59.711162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:46:59.711191 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:46:59.711622 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.711635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.713155 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.713161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.715321 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:46:59.715338 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.717193 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.717200 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:46:59.717380 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.717419 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:46:59.718307 (+ 888us) client_negotiation.cc:240] Negotiation successful
0925 02:46:59.718494 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6941,"thread_start_us":6732,"threads_started":1}
I20260925 02:46:59.719043 18927 raft_consensus.cc:2781] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:46:59.719726 18927 raft_consensus.cc:686] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 1 LEADER]: Becoming Leader. State: Replica: eb42d44fb6de4b34a056e8762ec49eb3, State: Running, Role: LEADER
I20260925 02:46:59.720342 18927 consensus_queue.cc:237] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43305 } } peers { permanent_uuid: "9793c0aa22c74e8ca16d7f60038443f5" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34019 } } peers { permanent_uuid: "97c83d8b2dde4fa4a844a00e6def068a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46811 } }
I20260925 02:46:59.720546 18567 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4358076c28254040a8e6ebb4ab9db147" candidate_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c83d8b2dde4fa4a844a00e6def068a" is_pre_election: true
I20260925 02:46:59.721436 18567 raft_consensus.cc:2454] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb42d44fb6de4b34a056e8762ec49eb3 in term 0.
I20260925 02:46:59.722030 18566 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4358076c28254040a8e6ebb4ab9db147" candidate_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97c83d8b2dde4fa4a844a00e6def068a"
I20260925 02:46:59.722313 18566 raft_consensus.cc:3037] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:46:59.726181 18566 raft_consensus.cc:2454] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb42d44fb6de4b34a056e8762ec49eb3 in term 1.
I20260925 02:46:59.729705 18411 catalog_manager.cc:5495] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb42d44fb6de4b34a056e8762ec49eb3 (127.5.56.130). New cstate: current_term: 1 leader_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb42d44fb6de4b34a056e8762ec49eb3" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9793c0aa22c74e8ca16d7f60038443f5" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97c83d8b2dde4fa4a844a00e6def068a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46811 } health_report { overall_health: UNKNOWN } } }
I20260925 02:46:59.746841 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.741587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54410 (local address 127.5.56.130:43305)
0925 02:46:59.741766 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:59.741771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:59.741788 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:46:59.741949 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:59.741952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:59.742053 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:59.742179 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:46:59.742646 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.743366 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.744815 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.745200 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.745303 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.745731 (+ 428us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:46:59.746263 (+ 532us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:46:59.746420 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:59.746490 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:46:59.746581 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:46:59.755151 18566 raft_consensus.cc:1261] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a [term 1 FOLLOWER]: Refusing update from remote peer eb42d44fb6de4b34a056e8762ec49eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:46:59.755190 18850 raft_consensus.cc:1261] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5 [term 1 FOLLOWER]: Refusing update from remote peer eb42d44fb6de4b34a056e8762ec49eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:46:59.756162 18926 consensus_queue.cc:1035] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9793c0aa22c74e8ca16d7f60038443f5" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 34019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:46:59.756593 18927 consensus_queue.cc:1035] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97c83d8b2dde4fa4a844a00e6def068a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:46:59.769456 18940 mvcc.cc:205] Tried to move back new op lower bound from 7333086903306067968 to 7333086903180279808. Current Snapshot: MvccSnapshot[applied={T|T < 7333086903306067968}]
I20260925 02:46:59.783186 18410 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:46:59.783528 18410 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34148:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:46:59.787452 18410 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 605bc8465e79412399a52e2fbd678dcd: Sending DeleteTablet for 3 replicas of tablet 4358076c28254040a8e6ebb4ab9db147
I20260925 02:46:59.788767 18694 tablet_service.cc:1514] Processing DeleteTablet for tablet 4358076c28254040a8e6ebb4ab9db147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:46:59 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:54408
I20260925 02:46:59.789041 18547 tablet_service.cc:1514] Processing DeleteTablet for tablet 4358076c28254040a8e6ebb4ab9db147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:46:59 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:41578
I20260925 02:46:59.789294 18830 tablet_service.cc:1514] Processing DeleteTablet for tablet 4358076c28254040a8e6ebb4ab9db147 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:46:59 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:40070
I20260925 02:46:59.794826 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 18449
I20260925 02:46:59.801194 18951 tablet_replica.cc:330] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5: stopping tablet replica
I20260925 02:46:59.801388 18949 tablet_replica.cc:330] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3: stopping tablet replica
I20260925 02:46:59.801884 18949 raft_consensus.cc:2229] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:46:59.801884 18951 raft_consensus.cc:2229] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:46:59.802523 18951 raft_consensus.cc:2258] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:46:59.802649 18949 raft_consensus.cc:2258] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:46:59.805122 18951 ts_tablet_manager.cc:1905] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:46:59.810117 18949 ts_tablet_manager.cc:1905] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:46:59.814862 18951 ts_tablet_manager.cc:1918] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:46:59.814932 18950 tablet_replica.cc:330] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a: stopping tablet replica
I20260925 02:46:59.815160 18951 log.cc:1196] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/4358076c28254040a8e6ebb4ab9db147
I20260925 02:46:59.815400 18950 raft_consensus.cc:2229] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:46:59.815888 18951 ts_tablet_manager.cc:1939] T 4358076c28254040a8e6ebb4ab9db147 P 9793c0aa22c74e8ca16d7f60038443f5: Deleting consensus metadata
I20260925 02:46:59.815973 18950 raft_consensus.cc:2258] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:46:59.816591 18917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.796411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.5.56.129:46811)
0925 02:46:59.796587 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:59.796594 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:59.796675 (+ 81us) server_negotiation.cc:407] Connection header received
0925 02:46:59.796895 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:59.796900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:59.796997 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:59.797089 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:59.800331 (+ 3242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.801734 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.804394 (+ 2660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.805105 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.805242 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.808838 (+ 3596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:59.808856 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:59.808875 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:59.808919 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:59.812860 (+ 3941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:59.813748 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:59.813760 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:59.813765 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:59.813982 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:59.814483 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:59.814491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:59.814498 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:59.815017 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:59.815338 (+ 321us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:59.815667 (+ 329us) server_negotiation.cc:299] Negotiation successful
0925 02:46:59.815923 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:46:59.817567 18609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:46:59.819357 18397 catalog_manager.cc:4841] TS 9793c0aa22c74e8ca16d7f60038443f5 (127.5.56.131:34019): tablet 4358076c28254040a8e6ebb4ab9db147 (table test-table [id=9d013f5098424752801687f847bcf6f8]) successfully deleted
I20260925 02:46:59.821672 18949 ts_tablet_manager.cc:1918] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:46:59.821915 18949 log.cc:1196] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/4358076c28254040a8e6ebb4ab9db147
I20260925 02:46:59.822562 18949 ts_tablet_manager.cc:1939] T 4358076c28254040a8e6ebb4ab9db147 P eb42d44fb6de4b34a056e8762ec49eb3: Deleting consensus metadata
I20260925 02:46:59.824702 18398 catalog_manager.cc:4841] TS eb42d44fb6de4b34a056e8762ec49eb3 (127.5.56.130:43305): tablet 4358076c28254040a8e6ebb4ab9db147 (table test-table [id=9d013f5098424752801687f847bcf6f8]) successfully deleted
I20260925 02:46:59.818430 18950 ts_tablet_manager.cc:1905] T 4358076c28254040a8e6ebb4ab9db147 P 97c83d8b2dde4fa4a844a00e6def068a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:46:59.910027 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 18449
I20260925 02:46:59.928802 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 18619
W20260925 02:46:59.928848 18398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.56.129:46811 (error 108)
I20260925 02:46:59.931730 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.931063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46811 (local address 127.0.0.1:41594)
0925 02:46:59.931373 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:59.931533 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
W20260925 02:46:59.932169 18398 catalog_manager.cc:4584] TS 97c83d8b2dde4fa4a844a00e6def068a (127.5.56.129:46811): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4358076c28254040a8e6ebb4ab9db147: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46811: connect: Connection refused (error 111)
I20260925 02:46:59.945766 18913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.929722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.5.56.130:43305)
0925 02:46:59.929957 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 02:46:59.929966 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:46:59.929994 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:46:59.930204 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:46:59.930212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:46:59.930343 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:46:59.930497 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:46:59.932502 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.933803 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.935201 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:46:59.935885 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:46:59.936020 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:46:59.938668 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:46:59.938686 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:46:59.938707 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:46:59.938769 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:46:59.942771 (+ 4002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:59.943421 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:59.943431 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:59.943435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:59.943607 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:46:59.943988 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:46:59.943994 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:46:59.943997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:46:59.944474 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:46:59.944809 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0925 02:46:59.944988 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 02:46:59.945350 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:46:59.946635 18755 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:46:59.986235 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.985662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46811 (local address 127.0.0.1:41606)
0925 02:46:59.985904 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:59.986031 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260925 02:46:59.987367 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:46:59.986980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46811 (local address 127.0.0.1:41622)
0925 02:46:59.987133 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:46:59.987221 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260925 02:47:00.047973 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:00.047376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46811 (local address 127.0.0.1:41632)
0925 02:47:00.047652 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:00.047765 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 02:47:00.049180 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:00.048804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46811 (local address 127.0.0.1:41644)
0925 02:47:00.048957 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:00.049041 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 02:47:00.062624 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 18619
I20260925 02:47:00.081964 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 18766
I20260925 02:47:00.095487 18915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:00.082838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.5.56.131:34019)
0925 02:47:00.083016 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:00.083020 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:00.083110 (+ 90us) server_negotiation.cc:407] Connection header received
0925 02:47:00.083335 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:00.083340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:00.083406 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:00.083491 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:00.084726 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:00.085443 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:00.086849 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:00.087287 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:00.087381 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:00.090018 (+ 2637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:00.090034 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:00.090051 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:00.090103 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:00.092824 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:00.093504 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:00.093511 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:00.093513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:00.093636 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:00.093966 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:00.093969 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:00.093971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:00.094331 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:00.094642 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:00.094935 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 02:47:00.095108 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:47:00.096150 18890 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:00.129868 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:00.129361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46811 (local address 127.0.0.1:41654)
0925 02:47:00.129604 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:00.129705 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260925 02:47:00.130940 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:00.130589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46811 (local address 127.0.0.1:41658)
0925 02:47:00.130764 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:00.130837 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 02:47:00.186869 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 18766
I20260925 02:47:00.204412 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 18377
I20260925 02:47:00.205565 18442 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:00.319890 18916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:00.319012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46811 (local address 127.0.0.1:41674)
0925 02:47:00.319237 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:00.319353 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46811: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 02:47:00.320017 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 18377
I20260925 02:47:00.343446 5346 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 02:47:00.689729 5346 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 02:47:00.689966 5346 mini_ranger.cc:66] Stopping Ranger...
I20260925 02:47:01.036113 5346 mini_ranger.cc:68] Stopped Ranger
2026-09-25 02:47:01.036 UTC [17414] LOG: received smart shutdown request
2026-09-25 02:47:01.039 UTC [17414] LOG: background worker "logical replication launcher" (PID 17424) exited with exit code 1
2026-09-25 02:47:01.039 UTC [17418] LOG: shutting down
2026-09-25 02:47:01.213 UTC [17414] LOG: database system is shut down
2026-09-25T02:47:01Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52955 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:47:01 dist-test-slave-n563 krb5kdc[18965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:47:01 dist-test-slave-n563 krb5kdc[18965](info): set up 2 sockets
Sep 25 02:47:01 dist-test-slave-n563 krb5kdc[18965](info): commencing operation
krb5kdc: starting...
W20260925 02:47:02.459993 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:47:02 dist-test-slave-n563 krb5kdc[18965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:47:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:47:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:47:02.708385 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:39353
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:39353,127.5.56.189:44653,127.5.56.188:39327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:03.050899 18978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:03.051218 18978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:03.051314 18978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:03.058525 18978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:03.058642 18978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:03.058686 18978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:03.058765 18978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:03.058810 18978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:03.068988 18978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:39353,127.5.56.189:44653,127.5.56.188:39327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:39353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:03.070374 18978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:03.071662 18978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:03.084281 18983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:03.084383 18984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:03.084334 18986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:03.087869 18978 server_base.cc:1034] running on GCE node
I20260925 02:47:03.088740 18978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:03.090309 18978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:03.091593 18978 hybrid_clock.cc:648] HybridClock initialized: now 1790304423091546 us; error 36 us; skew 500 ppm
I20260925 02:47:03.091998 18978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:03 dist-test-slave-n563 krb5kdc[18965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304423, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:03.095166 18978 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:47:03.096796 18978 webserver.cc:460] Webserver started at http://127.5.56.190:41365/ using document root <none> and password file <none>
I20260925 02:47:03.097575 18978 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:03.097728 18978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:03.098052 18978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:03.101189 18978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "f517cf6dd9264077b73908309f088cae"
format_stamp: "Formatted at 2026-09-25 02:47:03 on dist-test-slave-n563"
server_key: "ef2981c889d35d2a63e46b0260e97f7d"
server_key_iv: "342eeb8cac846f6cbdd00c3e7ddce983"
server_key_version: "encryptionkey@0"
I20260925 02:47:03.102098 18978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "f517cf6dd9264077b73908309f088cae"
format_stamp: "Formatted at 2026-09-25 02:47:03 on dist-test-slave-n563"
server_key: "ef2981c889d35d2a63e46b0260e97f7d"
server_key_iv: "342eeb8cac846f6cbdd00c3e7ddce983"
server_key_version: "encryptionkey@0"
I20260925 02:47:03.107498 18978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 02:47:03.111016 18993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:03.112432 18978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260925 02:47:03.112612 18978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "f517cf6dd9264077b73908309f088cae"
format_stamp: "Formatted at 2026-09-25 02:47:03 on dist-test-slave-n563"
server_key: "ef2981c889d35d2a63e46b0260e97f7d"
server_key_iv: "342eeb8cac846f6cbdd00c3e7ddce983"
server_key_version: "encryptionkey@0"
I20260925 02:47:03.112829 18978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:03.139667 18978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:03.143992 18978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:03.144310 18978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:03.163810 18978 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:39353
I20260925 02:47:03.163843 19045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:39353 every 8 connection(s)
I20260925 02:47:03.165268 18978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:47:03.169128 19047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:03.173734 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 18978
I20260925 02:47:03.173996 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:47:03.174867 5346 external_mini_cluster.cc:1428] Setting key c503abe2a3f9770049ce41284ac35557
I20260925 02:47:03.177201 19047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:47:03 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304423, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.56.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.189@KRBTEST.COM" created.
I20260925 02:47:03.203042 19050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.182835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.5.56.190:39353)
0925 02:47:03.183630 (+ 795us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:03.183640 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:03.183701 (+ 61us) server_negotiation.cc:407] Connection header received
0925 02:47:03.185099 (+ 1398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:03.185105 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:03.185353 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:03.185525 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:03.187194 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.188428 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.190781 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.191367 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.191509 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:03.195820 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:03.195828 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:03.195841 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:03.195876 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:03.198737 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:03.199523 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:03.199529 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:03.199531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:03.199677 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:03.200127 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:03.200130 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:03.200133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:03.200688 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:03.200869 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:03.201471 (+ 602us) server_negotiation.cc:299] Negotiation successful
0925 02:47:03.202015 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":171,"server-negotiator.queue_time_us":548,"thread_start_us":360,"threads_started":1}
I20260925 02:47:03.203136 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.181321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:39353 (local address 127.0.0.1:42830)
0925 02:47:03.182310 (+ 989us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.182364 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:03.184681 (+ 2317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:03.185595 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:03.185605 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:03.186469 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:03.186982 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:03.187011 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.188593 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.188602 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:03.190628 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:03.190646 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.191608 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.191621 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:03.191858 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:03.192598 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:03.192634 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:03.195580 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:03.198883 (+ 3303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:03.198895 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:03.198908 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:03.199380 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:03.199790 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:03.199796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:03.199799 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:03.199993 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:03.200824 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:03.200831 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:03.201078 (+ 247us) client_negotiation.cc:769] Sending connection context
0925 02:47:03.201310 (+ 232us) client_negotiation.cc:240] Negotiation successful
0925 02:47:03.201806 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"thread_start_us":472,"threads_started":1}
I20260925 02:47:03.206302 19047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:47:03.207605 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.207088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39322)
0925 02:47:03.207302 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.207435 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260925 02:47:03.207942 18996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
I20260925 02:47:03.209738 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.209253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39328)
0925 02:47:03.209445 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.209543 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:47:03.209957 19047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:44653: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab.
Entry for principal kudu/127.5.56.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab.
I20260925 02:47:03.231837 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.189:44653
--webserver_interface=127.5.56.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab
--principal=kudu/127.5.56.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:39353,127.5.56.189:44653,127.5.56.188:39327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:47:03.259363 19047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } attempt: 1
I20260925 02:47:03.260622 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.260102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39334)
0925 02:47:03.260327 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.260440 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 02:47:03.261690 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.261276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39346)
0925 02:47:03.261426 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.261507 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:47:03.261888 19047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:44653: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
I20260925 02:47:03.330181 19047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } attempt: 2
I20260925 02:47:03.331501 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.330948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39354)
0925 02:47:03.331181 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.331295 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 02:47:03.332640 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.332129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39360)
0925 02:47:03.332302 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.332420 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 02:47:03.332854 19047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:44653: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
I20260925 02:47:03.424146 19047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } attempt: 3
I20260925 02:47:03.425438 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.424901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39364)
0925 02:47:03.425141 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.425255 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260925 02:47:03.426582 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.426103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39380)
0925 02:47:03.426253 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.426364 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:47:03.426765 19047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:44653: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
I20260925 02:47:03.570026 19047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } attempt: 4
I20260925 02:47:03.571424 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.570769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39384)
0925 02:47:03.571132 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.571243 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181}
I20260925 02:47:03.572549 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.572117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39386)
0925 02:47:03.572288 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.572401 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 02:47:03.572805 19047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:44653: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44653: connect: Connection refused (error 111)
W20260925 02:47:03.576141 19054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:03.576463 19054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:03.576627 19054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:03.584270 19054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:03.584411 19054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:03.584517 19054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:03.584626 19054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:03.584695 19054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:03.595052 19054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:39353,127.5.56.189:44653,127.5.56.188:39327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab
--principal=kudu/127.5.56.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.189:44653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.56.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:03.596433 19054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:03.597711 19054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:03.610184 19059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:03.610224 19060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:03.610502 19062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:03.610976 19054 server_base.cc:1034] running on GCE node
I20260925 02:47:03.611901 19054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:03.613407 19054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:03.614607 19054 hybrid_clock.cc:648] HybridClock initialized: now 1790304423614553 us; error 45 us; skew 500 ppm
I20260925 02:47:03.614979 19054 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:03 dist-test-slave-n563 krb5kdc[18965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304423, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:03.622068 19054 init.cc:377] Logged in from keytab as kudu/127.5.56.189@KRBTEST.COM (short username kudu)
I20260925 02:47:03.623863 19054 webserver.cc:460] Webserver started at http://127.5.56.189:40119/ using document root <none> and password file <none>
I20260925 02:47:03.624622 19054 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:03.624758 19054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:03.625088 19054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:03.628269 19054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/data/instance:
uuid: "2cf9b4eef28349929f5d81bbe585e62f"
format_stamp: "Formatted at 2026-09-25 02:47:03 on dist-test-slave-n563"
server_key: "3a677f4361a8daeda650cf4e382d7012"
server_key_iv: "91533b3c3a1c34079f830ad95853995b"
server_key_version: "encryptionkey@0"
I20260925 02:47:03.629125 19054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/wal/instance:
uuid: "2cf9b4eef28349929f5d81bbe585e62f"
format_stamp: "Formatted at 2026-09-25 02:47:03 on dist-test-slave-n563"
server_key: "3a677f4361a8daeda650cf4e382d7012"
server_key_iv: "91533b3c3a1c34079f830ad95853995b"
server_key_version: "encryptionkey@0"
I20260925 02:47:03.634634 19054 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:47:03.638079 19069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:03.639536 19054 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:03.639731 19054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/wal
uuid: "2cf9b4eef28349929f5d81bbe585e62f"
format_stamp: "Formatted at 2026-09-25 02:47:03 on dist-test-slave-n563"
server_key: "3a677f4361a8daeda650cf4e382d7012"
server_key_iv: "91533b3c3a1c34079f830ad95853995b"
server_key_version: "encryptionkey@0"
I20260925 02:47:03.639937 19054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:03.666759 19054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:03.671022 19054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:03.671334 19054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:03.690717 19054 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.189:44653
I20260925 02:47:03.690768 19121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.189:44653 every 8 connection(s)
I20260925 02:47:03.692119 19054 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
I20260925 02:47:03.695920 19122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:03.697058 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 19054
I20260925 02:47:03.697273 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-1/wal/instance
I20260925 02:47:03.697860 5346 external_mini_cluster.cc:1428] Setting key 104d55694b82f0c78c7ae56412075a38
I20260925 02:47:03.706135 19122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:47:03 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304423, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.56.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.188@KRBTEST.COM" created.
I20260925 02:47:03.728454 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.709297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42834 (local address 127.5.56.190:39353)
0925 02:47:03.709793 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:03.709798 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:03.710967 (+ 1169us) server_negotiation.cc:407] Connection header received
0925 02:47:03.712865 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:03.712871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:03.712964 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:03.713089 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:03.714737 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.715473 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.717224 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.717640 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.717803 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:03.721644 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:03.721652 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:03.721657 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:03.721716 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:03.724740 (+ 3024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:03.725529 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:03.725535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:03.725545 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:03.725670 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:03.726224 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:03.726232 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:03.726237 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:03.726728 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:03.726921 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:03.727813 (+ 892us) server_negotiation.cc:299] Negotiation successful
0925 02:47:03.728048 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":282,"threads_started":1}
I20260925 02:47:03.729686 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.709930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:39353 (local address 127.0.0.1:42834)
0925 02:47:03.710800 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.710842 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:03.712577 (+ 1735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:03.713346 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:03.713356 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:03.713889 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:03.714556 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:03.714576 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.715674 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.715679 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:03.717072 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:03.717086 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.717885 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.717892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:03.718125 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:03.718861 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:03.718886 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:03.721357 (+ 2471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:03.724895 (+ 3538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:03.724905 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:03.724925 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:03.725393 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:03.725786 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:03.725791 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:03.725795 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:03.726066 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:03.727009 (+ 943us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:03.727018 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:03.727427 (+ 409us) client_negotiation.cc:769] Sending connection context
0925 02:47:03.727717 (+ 290us) client_negotiation.cc:240] Negotiation successful
0925 02:47:03.728285 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":377,"threads_started":1}
I20260925 02:47:03.733184 19122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:47:03 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304423, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
I20260925 02:47:03.749595 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.734046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39388)
0925 02:47:03.734249 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.734272 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:03.734425 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:03.735632 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:03.735636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:03.735999 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:03.736139 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:03.736147 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.738193 (+ 2046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.738197 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:03.739104 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:03.739116 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.740385 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.740389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:03.740546 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:03.741190 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:03.741209 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:03.743615 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:03.746968 (+ 3353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:03.746975 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:03.746982 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:03.747361 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:03.747728 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:03.747734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:03.747738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:03.747890 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:03.748541 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:03.748546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:03.748687 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 02:47:03.748836 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 02:47:03.749127 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 02:47:03.749639 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.734215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39388 (local address 127.5.56.189:44653)
0925 02:47:03.735042 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:03.735053 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:03.735109 (+ 56us) server_negotiation.cc:407] Connection header received
0925 02:47:03.735173 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:03.735185 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:03.735461 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:03.735630 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:03.736298 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.738051 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.739247 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.740238 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.740380 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:03.743846 (+ 3466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:03.743858 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:03.743873 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:03.743902 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:03.746835 (+ 2933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:03.747505 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:03.747512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:03.747514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:03.747619 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:03.748001 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:03.748006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:03.748009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:03.748418 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:03.748640 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:03.749000 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 02:47:03.749237 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":626,"thread_start_us":472,"threads_started":1}
I20260925 02:47:03.752287 19122 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:47:03.753509 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.753028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:51994)
0925 02:47:03.753199 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.753360 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 02:47:03.753749 19070 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
I20260925 02:47:03.755401 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.755041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52004)
0925 02:47:03.755215 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.755289 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260925 02:47:03.755636 19122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:39327: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab.
Entry for principal kudu/127.5.56.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab.
I20260925 02:47:03.758044 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.188:39327
--webserver_interface=127.5.56.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab
--principal=kudu/127.5.56.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:34939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:39353,127.5.56.189:44653,127.5.56.188:39327
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:47:03.804937 19122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } attempt: 1
I20260925 02:47:03.806445 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.805805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52014)
0925 02:47:03.806077 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.806214 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260925 02:47:03.807718 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.807287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52020)
0925 02:47:03.807492 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.807573 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260925 02:47:03.807963 19122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:39327: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
I20260925 02:47:03.872152 19047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } attempt: 5
I20260925 02:47:03.876183 19122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } attempt: 2
I20260925 02:47:03.877315 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.876839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52024)
0925 02:47:03.877033 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.877149 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260925 02:47:03.878605 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.878178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52040)
0925 02:47:03.878351 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.878459 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260925 02:47:03.879354 19122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:39327: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Sep 25 02:47:03 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304423, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
I20260925 02:47:03.888718 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.873369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39396 (local address 127.5.56.189:44653)
0925 02:47:03.873573 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:03.873578 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:03.873605 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:47:03.873691 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:03.873695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:03.873764 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:03.873866 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:03.874920 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.875838 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.877563 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.878356 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.878544 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:03.882350 (+ 3806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:03.882356 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:03.882360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:03.882452 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:03.885521 (+ 3069us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:03.886375 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:03.886404 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:03.886409 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:03.886564 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:03.887011 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:03.887019 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:03.887023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:03.887375 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:03.887546 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:03.888043 (+ 497us) server_negotiation.cc:299] Negotiation successful
0925 02:47:03.888297 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 02:47:03.888737 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.873046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39396)
0925 02:47:03.873335 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.873362 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:03.873518 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:03.873938 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:03.873946 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:03.874517 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:03.874748 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:03.874761 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.876016 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.876022 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:03.877378 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:03.877393 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:03.878560 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:03.878567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:03.878877 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:03.879628 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:03.879650 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:03.882103 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:03.885736 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:03.885742 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:03.885748 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:03.886217 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:03.886701 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:03.886707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:03.886712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:03.886883 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:03.887524 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:03.887530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:03.887730 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 02:47:03.887911 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 02:47:03.888274 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 02:47:03.890112 19047 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:47:03.891314 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.890907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52048)
0925 02:47:03.891088 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.891180 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 02:47:03.892338 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.892038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52050)
0925 02:47:03.892167 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.892237 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:47:03.892578 19047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:39327: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
I20260925 02:47:03.943813 19047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } attempt: 1
I20260925 02:47:03.945158 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.944629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52058)
0925 02:47:03.944895 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.944988 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 02:47:03.946118 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.945796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52066)
0925 02:47:03.945937 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.946013 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 02:47:03.946363 19047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:39327: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
I20260925 02:47:03.970588 19122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } attempt: 3
I20260925 02:47:03.971936 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.971400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52068)
0925 02:47:03.971656 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.971771 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260925 02:47:03.973140 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:03.972682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52070)
0925 02:47:03.972923 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:03.973026 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141}
W20260925 02:47:03.973374 19122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:39327: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
I20260925 02:47:04.014662 19047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } attempt: 2
I20260925 02:47:04.016047 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.015353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52086)
0925 02:47:04.015663 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:04.015793 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260925 02:47:04.017292 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.016844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52092)
0925 02:47:04.017021 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:04.017128 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 02:47:04.017526 19047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:39327: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
W20260925 02:47:04.111536 19130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:04.111842 19130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:04.111931 19130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 02:47:04.116643 19122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } attempt: 4
I20260925 02:47:04.118032 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.117409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52098)
0925 02:47:04.117669 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:04.117871 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 02:47:04.119128 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.118779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52108)
0925 02:47:04.118951 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:04.119026 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260925 02:47:04.119379 19122 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:39327: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
W20260925 02:47:04.120019 19130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:04.120158 19130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:04.120262 19130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:04.120343 19130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:04.120409 19130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:04.123757 19047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } attempt: 3
I20260925 02:47:04.124835 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.124355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52118)
0925 02:47:04.124559 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:04.124655 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 02:47:04.125901 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.125491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52120)
0925 02:47:04.125700 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:04.125780 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260925 02:47:04.126130 19047 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:39327: Network error: Client connection negotiation failed: client connection to 127.5.56.188:39327: connect: Connection refused (error 111)
I20260925 02:47:04.131755 19130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:34939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:39353,127.5.56.189:44653,127.5.56.188:39327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab
--principal=kudu/127.5.56.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.188:39327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.56.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:04.133104 19130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:04.134346 19130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:04.147051 19138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:04.147078 19136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:04.147344 19135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:04.148218 19130 server_base.cc:1034] running on GCE node
I20260925 02:47:04.149034 19130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:04.150636 19130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:04.151875 19130 hybrid_clock.cc:648] HybridClock initialized: now 1790304424151810 us; error 61 us; skew 500 ppm
I20260925 02:47:04.152236 19130 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:04 dist-test-slave-n563 krb5kdc[18965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304424, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:04.155534 19130 init.cc:377] Logged in from keytab as kudu/127.5.56.188@KRBTEST.COM (short username kudu)
I20260925 02:47:04.157038 19130 webserver.cc:460] Webserver started at http://127.5.56.188:43261/ using document root <none> and password file <none>
I20260925 02:47:04.157809 19130 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:04.157965 19130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:04.158286 19130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:04.161428 19130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/data/instance:
uuid: "6c637e40b1314bd6b53c398f9cff54c4"
format_stamp: "Formatted at 2026-09-25 02:47:04 on dist-test-slave-n563"
server_key: "f610cc0de23421a708eec885ff0fd7c1"
server_key_iv: "7f1b1bcee6125265ffdc2bc967d4477c"
server_key_version: "encryptionkey@0"
I20260925 02:47:04.162274 19130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/wal/instance:
uuid: "6c637e40b1314bd6b53c398f9cff54c4"
format_stamp: "Formatted at 2026-09-25 02:47:04 on dist-test-slave-n563"
server_key: "f610cc0de23421a708eec885ff0fd7c1"
server_key_iv: "7f1b1bcee6125265ffdc2bc967d4477c"
server_key_version: "encryptionkey@0"
I20260925 02:47:04.167907 19130 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:47:04.171633 19145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:04.173064 19130 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:47:04.173271 19130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/wal
uuid: "6c637e40b1314bd6b53c398f9cff54c4"
format_stamp: "Formatted at 2026-09-25 02:47:04 on dist-test-slave-n563"
server_key: "f610cc0de23421a708eec885ff0fd7c1"
server_key_iv: "7f1b1bcee6125265ffdc2bc967d4477c"
server_key_version: "encryptionkey@0"
I20260925 02:47:04.173496 19130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:04.189288 19130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:04.194084 19130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:04.194437 19130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:04.214056 19130 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.188:39327
I20260925 02:47:04.214079 19197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.188:39327 every 8 connection(s)
I20260925 02:47:04.215555 19130 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
I20260925 02:47:04.219772 19198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:04.222672 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 19130
I20260925 02:47:04.222898 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790304212743187-5346-0/minicluster-data/master-2/wal/instance
I20260925 02:47:04.223546 5346 external_mini_cluster.cc:1428] Setting key dc3ae627c81e0b8d22c4e2afd525fdeb
I20260925 02:47:04.228051 19198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:47:04 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
Sep 25 02:47:04 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304424, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:04.246958 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.230861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.5.56.190:39353)
0925 02:47:04.231391 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:04.231398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:04.231925 (+ 527us) server_negotiation.cc:407] Connection header received
0925 02:47:04.232103 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:04.232108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:04.232182 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:04.232300 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:04.233693 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.234535 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.236274 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.236656 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.236757 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.240464 (+ 3707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:04.240469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:04.240472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:04.240515 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:04.243807 (+ 3292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.244833 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.244836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.244839 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.244965 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.245356 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.245359 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.245361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.245680 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:04.245839 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:04.246428 (+ 589us) server_negotiation.cc:299] Negotiation successful
0925 02:47:04.246670 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":251,"threads_started":1}
I20260925 02:47:04.248154 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.230889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:39353 (local address 127.0.0.1:42840)
0925 02:47:04.231771 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:04.231822 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:04.231973 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:04.232360 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:04.232365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:04.233174 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:04.233522 (+ 348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:04.233537 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.234689 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.234693 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:04.236119 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:04.236135 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.236905 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.236914 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:04.237114 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.237757 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:04.237791 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:04.240266 (+ 2475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:04.243947 (+ 3681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:04.243956 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:04.243993 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:04.244708 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:04.245074 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:04.245079 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:04.245085 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:04.245253 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:04.245825 (+ 572us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:04.245831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:04.246068 (+ 237us) client_negotiation.cc:769] Sending connection context
0925 02:47:04.246266 (+ 198us) client_negotiation.cc:240] Negotiation successful
0925 02:47:04.246787 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"mutex_wait_us":157,"thread_start_us":485,"threads_started":1}
I20260925 02:47:04.248173 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.225530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52128 (local address 127.5.56.188:39327)
0925 02:47:04.226483 (+ 953us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:04.226495 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:04.226538 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:47:04.228196 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:04.228218 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:04.228623 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:04.229040 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:04.230676 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.232493 (+ 1817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.234302 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.235193 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.235329 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.240138 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:04.240151 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:04.240169 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:04.240258 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:04.243129 (+ 2871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.244029 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.244040 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.244048 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.244265 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.244681 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.244687 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.244692 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.245438 (+ 746us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:04.245658 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:04.246148 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 02:47:04.246666 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":682,"spinlock_wait_cycles":24320,"thread_start_us":482,"threads_started":1}
I20260925 02:47:04.249992 19198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:47:04 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304424, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
Sep 25 02:47:04 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:04.265192 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.250846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44653 (local address 127.0.0.1:39412)
0925 02:47:04.251039 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:04.251058 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:04.251206 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:04.251547 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:04.251552 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:04.251992 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:04.252194 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:04.252205 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.253691 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.253695 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:04.255055 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:04.255070 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.256014 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.256018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:04.256154 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.256833 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:04.256849 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:04.259511 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:04.262719 (+ 3208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:04.262725 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:04.262729 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:04.263108 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:04.263513 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:04.263519 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:04.263523 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:04.263681 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:04.264213 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:04.264218 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:04.264375 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 02:47:04.264520 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 02:47:04.264817 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 02:47:04.265211 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.250835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.5.56.189:44653)
0925 02:47:04.251043 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:04.251050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:04.251140 (+ 90us) server_negotiation.cc:407] Connection header received
0925 02:47:04.251340 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:04.251348 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:04.251417 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:04.251527 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:04.252375 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.253544 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.255197 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.255871 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.256006 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.259727 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:04.259732 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:04.259736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:04.259769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:04.262574 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.263254 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.263263 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.263266 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.263396 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.263799 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.263806 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.263809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.264065 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:04.264255 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:04.264629 (+ 374us) server_negotiation.cc:299] Negotiation successful
0925 02:47:04.264836 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:47:04.266355 19198 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:47:04.267863 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.251876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42844 (local address 127.5.56.190:39353)
0925 02:47:04.252079 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:04.252084 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:04.252102 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:47:04.252155 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:04.252159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:04.252240 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:04.252346 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:04.253933 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.255032 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.256702 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.257146 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.257242 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.261978 (+ 4736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:04.261983 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:04.261986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:04.262019 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:04.264693 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.265433 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.265441 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.265447 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.265592 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.266180 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.266185 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.266188 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.266609 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:04.266854 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:04.267284 (+ 430us) server_negotiation.cc:299] Negotiation successful
0925 02:47:04.267543 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
Sep 25 02:47:04 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304424, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
Sep 25 02:47:04 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
I20260925 02:47:04.281198 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.267190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52130)
0925 02:47:04.267384 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:04.267409 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:04.267512 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:04.267896 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:04.267899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:04.268315 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:04.268486 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:04.268500 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.269563 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.269568 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:04.270770 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:04.270781 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.271724 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.271729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:04.271957 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.272517 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:04.272539 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:04.274954 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:04.278784 (+ 3830us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:04.278787 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:04.278790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:04.279133 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:04.279559 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:04.279563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:04.279566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:04.279694 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:04.280350 (+ 656us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:04.280355 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:04.280509 (+ 154us) client_negotiation.cc:769] Sending connection context
0925 02:47:04.280641 (+ 132us) client_negotiation.cc:240] Negotiation successful
0925 02:47:04.280887 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 02:47:04.281536 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.267455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52130 (local address 127.5.56.188:39327)
0925 02:47:04.267604 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:04.267610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:04.267626 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:47:04.267673 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:04.267678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:04.267745 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:04.267858 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:04.268638 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.269436 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.270911 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.271583 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.271704 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.275143 (+ 3439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:04.275150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:04.275155 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:04.275194 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:04.278646 (+ 3452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.279268 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.279273 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.279276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.279437 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.279816 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.279824 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.279829 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.280203 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:04.280373 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:04.280769 (+ 396us) server_negotiation.cc:299] Negotiation successful
0925 02:47:04.281091 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":56}
I20260925 02:47:04.286691 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.269768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39414 (local address 127.5.56.189:44653)
0925 02:47:04.269933 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:04.269940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:04.269958 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:47:04.270064 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:04.270069 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:04.270136 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:04.270244 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:04.271714 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.272754 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.274225 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.274882 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.275026 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.280492 (+ 5466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:04.280497 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:04.280500 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:04.280532 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:04.283919 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.284676 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.284682 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.284685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.284854 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.285245 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.285251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.285254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.285613 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:04.285916 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:04.286093 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 02:47:04.286329 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:47:04.287910 19198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4: Bootstrap starting.
I20260925 02:47:04.291882 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:47:04.291900 19198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:04.292953 19198 log.cc:824] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:04.297751 19198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4: No bootstrap required, opened a new log
I20260925 02:47:04.303381 19047 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } attempt: 4
I20260925 02:47:04.313916 19198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } }
I20260925 02:47:04.316145 19198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:04.316586 19198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c637e40b1314bd6b53c398f9cff54c4, State: Initialized, Role: FOLLOWER
I20260925 02:47:04.319747 19198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } }
I20260925 02:47:04.320426 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.297167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39424 (local address 127.5.56.189:44653)
0925 02:47:04.297434 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:04.297440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:04.298230 (+ 790us) server_negotiation.cc:407] Connection header received
0925 02:47:04.298553 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:04.298560 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:04.298654 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:04.298734 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:04.301801 (+ 3067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.303389 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.305663 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.306611 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.306739 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.311625 (+ 4886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:04.311635 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:04.311643 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:04.311698 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:04.316018 (+ 4320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.317510 (+ 1492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.317519 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.317526 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.317732 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.318299 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.318304 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.318308 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.318701 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:04.318974 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:04.319734 (+ 760us) server_negotiation.cc:299] Negotiation successful
0925 02:47:04.320016 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 25 02:47:04 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304423, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
I20260925 02:47:04.326977 19213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } } }
I20260925 02:47:04.327430 19213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:47:04.329662 19198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:04.335537 19201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.298285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.5.56.188:39327)
0925 02:47:04.302658 (+ 4373us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:04.302667 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:04.315012 (+ 12345us) server_negotiation.cc:407] Connection header received
0925 02:47:04.315229 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:04.315235 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:04.315332 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:04.315434 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:04.317529 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.318782 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.320850 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.321608 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.321774 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.327618 (+ 5844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:04.327623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:04.327627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:04.327665 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:04.331843 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.332874 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.332880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.332884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.333060 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.333562 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.333569 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.333572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.333972 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:04.334198 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:04.334864 (+ 666us) server_negotiation.cc:299] Negotiation successful
0925 02:47:04.335113 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":4209}
I20260925 02:47:04.335729 19202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.297009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.5.56.190:39353)
0925 02:47:04.297939 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:04.297947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:04.315712 (+ 17765us) server_negotiation.cc:407] Connection header received
0925 02:47:04.316866 (+ 1154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:04.316872 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:04.316991 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:04.317073 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:04.320532 (+ 3459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.321553 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.323479 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.324154 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.324258 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.327921 (+ 3663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:04.327928 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:04.327932 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:04.327990 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:04.332306 (+ 4316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.333200 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.333205 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.333208 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.333380 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.333840 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.333845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.333848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.334210 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:04.334484 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:04.335082 (+ 598us) server_negotiation.cc:299] Negotiation successful
0925 02:47:04.335301 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":775}
I20260925 02:47:04.336886 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.303966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52138)
0925 02:47:04.304307 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:04.304338 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:04.304446 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:04.318800 (+ 14354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:04.318805 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:04.319334 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:04.319556 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:04.319573 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.322139 (+ 2566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.322144 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:04.323163 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:04.323176 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.324204 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.324213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:04.324412 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.325062 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:04.325094 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:04.328239 (+ 3145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:04.333095 (+ 4856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:04.333103 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:04.333111 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:04.333755 (+ 644us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:04.334333 (+ 578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:04.334341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:04.334347 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:04.334595 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:04.335439 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:04.335447 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:04.335701 (+ 254us) client_negotiation.cc:769] Sending connection context
0925 02:47:04.335914 (+ 213us) client_negotiation.cc:240] Negotiation successful
0925 02:47:04.336300 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"mutex_wait_us":1}
I20260925 02:47:04.339672 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.304646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52138 (local address 127.5.56.188:39327)
0925 02:47:04.318349 (+ 13703us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:04.318355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:04.318483 (+ 128us) server_negotiation.cc:407] Connection header received
0925 02:47:04.318538 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:04.318542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:04.318616 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:04.319707 (+ 1091us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:04.320018 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.322026 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.323329 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.324029 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.324708 (+ 679us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.328422 (+ 3714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:04.328428 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:04.328432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:04.328465 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:04.332886 (+ 4421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.333971 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.333979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.333985 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.334176 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.334793 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.334801 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.334807 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.335261 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:04.338586 (+ 3325us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:04.338763 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 02:47:04.339076 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":13541,"thread_start_us":13729,"threads_started":1}
I20260925 02:47:04.345003 19047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae: Bootstrap starting.
I20260925 02:47:04.348467 19047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:04.349623 19047 log.cc:824] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:04.354216 19047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae: No bootstrap required, opened a new log
W20260925 02:47:04.360359 19225 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:47:04.360580 19225 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:47:04.365710 19047 raft_consensus.cc:348] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } }
I20260925 02:47:04.366207 19047 raft_consensus.cc:374] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:04.366355 19047 raft_consensus.cc:729] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f517cf6dd9264077b73908309f088cae, State: Initialized, Role: FOLLOWER
I20260925 02:47:04.367051 19047 consensus_queue.cc:260] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } }
I20260925 02:47:04.356483 19130 master_runner.cc:186] Error getting master registration for 127.5.56.190:39353: OK, instance_id { permanent_uuid: "f517cf6dd9264077b73908309f088cae" instance_seqno: 1790304423158218 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:47:04.368144 19227 sys_catalog.cc:455] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } } }
I20260925 02:47:04.368593 19130 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:47:04.368665 19227 sys_catalog.cc:458] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:47:04.369385 19047 sys_catalog.cc:564] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [sys.catalog]: configured and running, proceeding with master startup.
W20260925 02:47:04.381564 19238 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:47:04.381784 19238 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:47:04.381187 18978 master_runner.cc:186] Error getting master registration for 127.5.56.189:44653: OK, instance_id { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" instance_seqno: 1790304423685214 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:47:04.389204 18978 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 02:47:04.397370 18999 tablet.cc:2369] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:04.418608 19122 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } attempt: 5
Sep 25 02:47:04 dist-test-slave-n563 krb5kdc[18965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304423, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
I20260925 02:47:04.434122 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.419353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52144 (local address 127.5.56.188:39327)
0925 02:47:04.419568 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:04.419575 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:04.419621 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:47:04.419842 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:04.419847 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:04.419925 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:04.420043 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:04.420846 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.421624 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.422973 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.423596 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.423712 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.428001 (+ 4289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:04.428014 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:04.428018 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:04.428048 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:04.431357 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.432065 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.432072 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.432076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.432223 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:04.432638 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:04.432645 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:04.432648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:04.432943 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:04.433137 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:04.433519 (+ 382us) server_negotiation.cc:299] Negotiation successful
0925 02:47:04.433740 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:47:04.434132 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:04.419284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:39327 (local address 127.0.0.1:52144)
0925 02:47:04.419537 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:04.419558 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:04.419705 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:04.420083 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:04.420088 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:04.420525 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:04.420688 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:04.420697 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.421771 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.421778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:04.422803 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:04.422825 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:04.423903 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:04.423911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:04.424148 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:04.425191 (+ 1043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:04.425215 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:04.427790 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:04.431503 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:04.431509 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:04.431513 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:04.431934 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:04.432344 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:04.432351 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:04.432355 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:04.432519 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:04.433075 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:04.433081 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:04.433243 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 02:47:04.433403 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 02:47:04.433718 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 02:47:04.439803 19122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f: Bootstrap starting.
I20260925 02:47:04.444715 19122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:04.445827 19122 log.cc:824] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f: Log is configured to *not* fsync() on all Append() calls
W20260925 02:47:04.446600 19151 tablet.cc:2369] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:04.448725 19122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f: No bootstrap required, opened a new log
I20260925 02:47:04.457561 19122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } }
I20260925 02:47:04.458030 19122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:04.458156 19122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cf9b4eef28349929f5d81bbe585e62f, State: Initialized, Role: FOLLOWER
I20260925 02:47:04.458865 19122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } }
I20260925 02:47:04.460042 19242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } } }
I20260925 02:47:04.460512 19242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:47:04.461341 19122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:04.485752 19054 master_runner.cc:263] No leader master found from master 2cf9b4eef28349929f5d81bbe585e62f
I20260925 02:47:04.485960 19054 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260925 02:47:04.489475 19253 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:47:04.489745 19253 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:47:04.537103 19242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:04.537489 19242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } }
I20260925 02:47:04.539521 19242 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f517cf6dd9264077b73908309f088cae (127.5.56.190:39353), 6c637e40b1314bd6b53c398f9cff54c4 (127.5.56.188:39327)
I20260925 02:47:04.540774 19173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cf9b4eef28349929f5d81bbe585e62f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c637e40b1314bd6b53c398f9cff54c4" is_pre_election: true
I20260925 02:47:04.541493 19173 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf9b4eef28349929f5d81bbe585e62f in term 0.
I20260925 02:47:04.542982 19020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cf9b4eef28349929f5d81bbe585e62f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f517cf6dd9264077b73908309f088cae" is_pre_election: true
I20260925 02:47:04.543659 19020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cf9b4eef28349929f5d81bbe585e62f in term 0.
I20260925 02:47:04.544611 19070 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cf9b4eef28349929f5d81bbe585e62f, 6c637e40b1314bd6b53c398f9cff54c4; no voters:
I20260925 02:47:04.549683 19242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:04.549865 19242 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:04.549968 19242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:04.553984 19242 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } }
I20260925 02:47:04.555366 19020 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cf9b4eef28349929f5d81bbe585e62f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f517cf6dd9264077b73908309f088cae"
I20260925 02:47:04.555611 19020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:04.559269 19020 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf9b4eef28349929f5d81bbe585e62f in term 1.
I20260925 02:47:04.559860 19073 leader_election.cc:302] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cf9b4eef28349929f5d81bbe585e62f, f517cf6dd9264077b73908309f088cae; no voters:
I20260925 02:47:04.562914 19173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2cf9b4eef28349929f5d81bbe585e62f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c637e40b1314bd6b53c398f9cff54c4"
I20260925 02:47:04.563216 19173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:04.566901 19173 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cf9b4eef28349929f5d81bbe585e62f in term 1.
I20260925 02:47:04.567966 19242 leader_election.cc:288] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [CANDIDATE]: Term 1 election: Requested vote from peers f517cf6dd9264077b73908309f088cae (127.5.56.190:39353), 6c637e40b1314bd6b53c398f9cff54c4 (127.5.56.188:39327)
I20260925 02:47:04.568372 19242 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:04.572414 19242 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [term 1 LEADER]: Becoming Leader. State: Replica: 2cf9b4eef28349929f5d81bbe585e62f, State: Running, Role: LEADER
I20260925 02:47:04.573350 19242 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } }
I20260925 02:47:04.587239 19255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf9b4eef28349929f5d81bbe585e62f. Latest consensus state: current_term: 1 leader_uuid: "2cf9b4eef28349929f5d81bbe585e62f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } } }
I20260925 02:47:04.587821 19255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:04.595155 19257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:47:04.597617 19257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:47:04.618361 19173 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [term 1 FOLLOWER]: Refusing update from remote peer 2cf9b4eef28349929f5d81bbe585e62f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:04.619860 19255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:04.627700 19020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [term 1 FOLLOWER]: Refusing update from remote peer 2cf9b4eef28349929f5d81bbe585e62f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:04.628772 19255 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:04.649745 19213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf9b4eef28349929f5d81bbe585e62f. Latest consensus state: current_term: 1 leader_uuid: "2cf9b4eef28349929f5d81bbe585e62f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } } }
I20260925 02:47:04.650123 19213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:47:04.655673 19258 mvcc.cc:205] Tried to move back new op lower bound from 7333086923224141824 to 7333086923065552896. Current Snapshot: MvccSnapshot[applied={T|T < 7333086923224141824}]
I20260925 02:47:04.657431 19227 sys_catalog.cc:455] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2cf9b4eef28349929f5d81bbe585e62f. Latest consensus state: current_term: 1 leader_uuid: "2cf9b4eef28349929f5d81bbe585e62f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } } }
I20260925 02:47:04.657811 19227 sys_catalog.cc:458] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:47:04.657717 19259 mvcc.cc:205] Tried to move back new op lower bound from 7333086923224141824 to 7333086923065552896. Current Snapshot: MvccSnapshot[applied={T|T < 7333086923224141824}]
I20260925 02:47:04.667775 19242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cf9b4eef28349929f5d81bbe585e62f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } } }
I20260925 02:47:04.668154 19242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:04.668638 19242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2cf9b4eef28349929f5d81bbe585e62f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } } }
I20260925 02:47:04.668892 19242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:04.670364 19257 catalog_manager.cc:1347] Generated new cluster ID: 2b0333ba7ca344bd8ea6db15b5932cdf
I20260925 02:47:04.670485 19257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:47:04.675915 19227 sys_catalog.cc:455] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cf9b4eef28349929f5d81bbe585e62f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } } }
I20260925 02:47:04.676211 19227 sys_catalog.cc:458] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:47:04.688668 19213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2cf9b4eef28349929f5d81bbe585e62f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f517cf6dd9264077b73908309f088cae" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 39353 } } peers { permanent_uuid: "2cf9b4eef28349929f5d81bbe585e62f" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44653 } } peers { permanent_uuid: "6c637e40b1314bd6b53c398f9cff54c4" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 39327 } } }
I20260925 02:47:04.688954 19213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:47:04.710312 19257 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:47:04.712218 19257 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:47:04.740774 19257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2cf9b4eef28349929f5d81bbe585e62f: Generated new TSK 0
I20260925 02:47:04.742281 19257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:47:05.362823 19225 catalog_manager.cc:1259] Loaded cluster ID: 2b0333ba7ca344bd8ea6db15b5932cdf
I20260925 02:47:05.362931 19225 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4: loading cluster ID for follower catalog manager: success
I20260925 02:47:05.365705 19225 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4: acquiring CA information for follower catalog manager: success
I20260925 02:47:05.367918 19225 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6c637e40b1314bd6b53c398f9cff54c4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:47:05.383898 19238 catalog_manager.cc:1259] Loaded cluster ID: 2b0333ba7ca344bd8ea6db15b5932cdf
I20260925 02:47:05.384052 19238 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae: loading cluster ID for follower catalog manager: success
I20260925 02:47:05.386590 19238 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae: acquiring CA information for follower catalog manager: success
I20260925 02:47:05.388495 19238 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f517cf6dd9264077b73908309f088cae: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:47:05.885213 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 18978
I20260925 02:47:05.886643 19041 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:06.000550 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 18978
I20260925 02:47:06.022568 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 19054
I20260925 02:47:06.023792 19117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:06.135123 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 19054
I20260925 02:47:06.157205 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 19130
I20260925 02:47:06.158510 19193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:06.268678 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 19130
2026-09-25T02:47:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4925 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:47:06 dist-test-slave-n563 krb5kdc[19276](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:47:06 dist-test-slave-n563 krb5kdc[19276](info): set up 2 sockets
Sep 25 02:47:06 dist-test-slave-n563 krb5kdc[19276](info): commencing operation
krb5kdc: starting...
W20260925 02:47:07.401124 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:47:07 dist-test-slave-n563 krb5kdc[19276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:47:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:47:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:47:07.659564 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41721
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:35281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:41721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:08.011124 19290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:08.011520 19290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:08.011746 19290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:08.019382 19290 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:08.019562 19290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:08.019701 19290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:08.019807 19290 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:08.019903 19290 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:08.030227 19290 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:35281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:41721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:08.031739 19290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:08.033202 19290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:08.045718 19295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:08.045758 19298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:08.045676 19296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:08.046759 19290 server_base.cc:1034] running on GCE node
I20260925 02:47:08.047544 19290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:08.048977 19290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:08.050271 19290 hybrid_clock.cc:648] HybridClock initialized: now 1790304428050214 us; error 62 us; skew 500 ppm
I20260925 02:47:08.050755 19290 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:08 dist-test-slave-n563 krb5kdc[19276](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304428, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:08.054090 19290 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:47:08.055812 19290 webserver.cc:460] Webserver started at http://127.5.56.190:37781/ using document root <none> and password file <none>
I20260925 02:47:08.056565 19290 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:08.056732 19290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:08.057070 19290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:08.060319 19290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "5aff3c122b2b44c9b4809724a7eb4e15"
format_stamp: "Formatted at 2026-09-25 02:47:08 on dist-test-slave-n563"
server_key: "de372eb81f41740cf36ee3e3005e778d"
server_key_iv: "f6aa5b75ef1dc5e7e0142b2b041e80d4"
server_key_version: "encryptionkey@0"
I20260925 02:47:08.061262 19290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "5aff3c122b2b44c9b4809724a7eb4e15"
format_stamp: "Formatted at 2026-09-25 02:47:08 on dist-test-slave-n563"
server_key: "de372eb81f41740cf36ee3e3005e778d"
server_key_iv: "f6aa5b75ef1dc5e7e0142b2b041e80d4"
server_key_version: "encryptionkey@0"
I20260925 02:47:08.066970 19290 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:47:08.070606 19306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:08.071982 19290 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:47:08.072189 19290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "5aff3c122b2b44c9b4809724a7eb4e15"
format_stamp: "Formatted at 2026-09-25 02:47:08 on dist-test-slave-n563"
server_key: "de372eb81f41740cf36ee3e3005e778d"
server_key_iv: "f6aa5b75ef1dc5e7e0142b2b041e80d4"
server_key_version: "encryptionkey@0"
I20260925 02:47:08.072412 19290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:08.094162 19290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:08.098574 19290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:08.098903 19290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:08.118712 19290 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:41721
I20260925 02:47:08.118760 19357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:41721 every 8 connection(s)
I20260925 02:47:08.120184 19290 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:47:08.124152 19358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:08.125103 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 19290
I20260925 02:47:08.125379 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:47:08.126248 5346 external_mini_cluster.cc:1428] Setting key f41d0492356b5e26d944c9c92a745da7
I20260925 02:47:08.137626 19358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15: Bootstrap starting.
Sep 25 02:47:08 dist-test-slave-n563 krb5kdc[19276](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:08.141032 19358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:08.142127 19358 log.cc:824] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:08.145313 19358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15: No bootstrap required, opened a new log
I20260925 02:47:08.149214 19361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:08.128336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.5.56.190:41721)
0925 02:47:08.129143 (+ 807us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:08.129166 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:08.129210 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:47:08.131112 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:08.131136 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:08.131552 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:08.131966 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:08.133523 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:08.134954 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:08.136324 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:08.137003 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:08.137225 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:08.141628 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:08.141636 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:08.141652 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:08.141694 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:08.144840 (+ 3146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:08.145680 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:08.145687 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:08.145693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:08.145897 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:08.146327 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:08.146332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:08.146336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:08.146920 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:08.147273 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:08.147733 (+ 460us) server_negotiation.cc:299] Negotiation successful
0925 02:47:08.148186 (+ 453us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":361,"threads_started":1}
I20260925 02:47:08.153388 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:47:08.153887 19358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aff3c122b2b44c9b4809724a7eb4e15" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41721 } }
I20260925 02:47:08.154260 19358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:08.154422 19358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aff3c122b2b44c9b4809724a7eb4e15, State: Initialized, Role: FOLLOWER
I20260925 02:47:08.155016 19358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aff3c122b2b44c9b4809724a7eb4e15" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41721 } }
I20260925 02:47:08.155259 19358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:47:08.155402 19358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:47:08.155583 19358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:08.158224 19358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aff3c122b2b44c9b4809724a7eb4e15" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41721 } }
I20260925 02:47:08.158823 19358 leader_election.cc:302] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5aff3c122b2b44c9b4809724a7eb4e15; no voters:
I20260925 02:47:08.159508 19358 leader_election.cc:288] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:47:08.159714 19363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:08.160526 19363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [term 1 LEADER]: Becoming Leader. State: Replica: 5aff3c122b2b44c9b4809724a7eb4e15, State: Running, Role: LEADER
I20260925 02:47:08.161079 19363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aff3c122b2b44c9b4809724a7eb4e15" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41721 } }
I20260925 02:47:08.161854 19358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:08.163846 19364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5aff3c122b2b44c9b4809724a7eb4e15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aff3c122b2b44c9b4809724a7eb4e15" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41721 } } }
I20260925 02:47:08.164177 19364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:08.165128 19369 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:47:08.164914 19365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5aff3c122b2b44c9b4809724a7eb4e15. Latest consensus state: current_term: 1 leader_uuid: "5aff3c122b2b44c9b4809724a7eb4e15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5aff3c122b2b44c9b4809724a7eb4e15" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41721 } } }
I20260925 02:47:08.166268 19365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:08.171010 19369 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:47:08.181666 19369 catalog_manager.cc:1347] Generated new cluster ID: ced61821d09549f994571e164fb6a9e7
I20260925 02:47:08.181802 19369 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:47:08.197770 19369 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:47:08.199762 19369 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:47:08.214226 19369 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5aff3c122b2b44c9b4809724a7eb4e15: Generated new TSK 0
I20260925 02:47:08.215433 19369 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:47:08.266366 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 19290
I20260925 02:47:08.267650 19353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:08.366792 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 19290
2026-09-25T02:47:08Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2093 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:47:08 dist-test-slave-n563 krb5kdc[19388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:47:08 dist-test-slave-n563 krb5kdc[19388](info): set up 2 sockets
Sep 25 02:47:08 dist-test-slave-n563 krb5kdc[19388](info): commencing operation
krb5kdc: starting...
W20260925 02:47:09.483979 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:47:09 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:47:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:47:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:47:09.742689 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41657
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:41657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:10.082049 19401 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:10.082446 19401 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:10.082631 19401 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:10.090088 19401 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:10.090270 19401 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:10.090368 19401 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:10.090502 19401 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:10.090592 19401 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:10.101171 19401 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:41657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:10.102746 19401 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:10.104106 19401 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:10.116838 19406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:10.116881 19409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:10.116912 19407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:10.117715 19401 server_base.cc:1034] running on GCE node
I20260925 02:47:10.118666 19401 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:10.120230 19401 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:10.121486 19401 hybrid_clock.cc:648] HybridClock initialized: now 1790304430121433 us; error 59 us; skew 500 ppm
I20260925 02:47:10.121868 19401 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:10 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304430, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:10.131819 19401 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:47:10.133477 19401 webserver.cc:460] Webserver started at http://127.5.56.190:37409/ using document root <none> and password file <none>
I20260925 02:47:10.134178 19401 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:10.134284 19401 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:10.134644 19401 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:10.137821 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "f67d111f1c2845a4a117f2cb83d5eb64"
format_stamp: "Formatted at 2026-09-25 02:47:10 on dist-test-slave-n563"
server_key: "5df4bf6bb4374b102932869e59529785"
server_key_iv: "8d508034d72501bb7097ef75d34ee2c5"
server_key_version: "encryptionkey@0"
I20260925 02:47:10.138670 19401 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "f67d111f1c2845a4a117f2cb83d5eb64"
format_stamp: "Formatted at 2026-09-25 02:47:10 on dist-test-slave-n563"
server_key: "5df4bf6bb4374b102932869e59529785"
server_key_iv: "8d508034d72501bb7097ef75d34ee2c5"
server_key_version: "encryptionkey@0"
I20260925 02:47:10.144158 19401 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:47:10.147718 19416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:10.149134 19401 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:47:10.149360 19401 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "f67d111f1c2845a4a117f2cb83d5eb64"
format_stamp: "Formatted at 2026-09-25 02:47:10 on dist-test-slave-n563"
server_key: "5df4bf6bb4374b102932869e59529785"
server_key_iv: "8d508034d72501bb7097ef75d34ee2c5"
server_key_version: "encryptionkey@0"
I20260925 02:47:10.149546 19401 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:10.189618 19401 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:10.194268 19401 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:10.194672 19401 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:10.214380 19401 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:41657
I20260925 02:47:10.214435 19468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:41657 every 8 connection(s)
I20260925 02:47:10.215929 19401 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:47:10.218361 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 19401
I20260925 02:47:10.218647 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:47:10.219560 5346 external_mini_cluster.cc:1428] Setting key 77de95419e1d613a0318acb47378bdaf
I20260925 02:47:10.220974 19469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:47:10 dist-test-slave-n563 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:10.237494 19469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Bootstrap starting.
I20260925 02:47:10.241353 19469 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:10.242607 19469 log.cc:824] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:10.244777 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:10.221859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38736 (local address 127.5.56.190:41657)
0925 02:47:10.222704 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:10.222716 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:10.222757 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:47:10.224381 (+ 1624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:10.224402 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:10.224825 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:10.225158 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:10.227232 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:10.228585 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:10.230622 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:10.231325 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:10.231465 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:10.237064 (+ 5599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:10.237072 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:10.237089 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:10.237128 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:10.240290 (+ 3162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:10.241219 (+ 929us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:10.241226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:10.241233 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:10.241409 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:10.241853 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:10.241856 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:10.241858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:10.242494 (+ 636us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:10.242874 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:10.243272 (+ 398us) server_negotiation.cc:299] Negotiation successful
0925 02:47:10.243745 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":368,"threads_started":1}
I20260925 02:47:10.246295 19469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: No bootstrap required, opened a new log
I20260925 02:47:10.255023 19469 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:10.255507 19469 raft_consensus.cc:374] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:10.255626 19469 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67d111f1c2845a4a117f2cb83d5eb64, State: Initialized, Role: FOLLOWER
I20260925 02:47:10.256244 19469 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:10.256516 19469 raft_consensus.cc:388] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:47:10.256678 19469 raft_consensus.cc:482] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:47:10.256865 19469 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:10.259627 19469 raft_consensus.cc:504] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:10.260191 19469 leader_election.cc:302] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f67d111f1c2845a4a117f2cb83d5eb64; no voters:
I20260925 02:47:10.260880 19469 leader_election.cc:288] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:47:10.261034 19474 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:10.261785 19474 raft_consensus.cc:686] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 1 LEADER]: Becoming Leader. State: Replica: f67d111f1c2845a4a117f2cb83d5eb64, State: Running, Role: LEADER
I20260925 02:47:10.262472 19474 consensus_queue.cc:237] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:10.263368 19469 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:10.265489 19476 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f67d111f1c2845a4a117f2cb83d5eb64. Latest consensus state: current_term: 1 leader_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } } }
I20260925 02:47:10.265986 19476 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:10.266014 19475 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } } }
I20260925 02:47:10.266959 19475 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:10.267668 19481 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:47:10.273039 19481 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:47:10.282949 19481 catalog_manager.cc:1347] Generated new cluster ID: d06b4eec85da4c08899127f75c6d45d7
I20260925 02:47:10.283071 19481 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:47:10.301254 19481 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:47:10.302645 19481 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:47:10.314129 19481 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Generated new TSK 0
I20260925 02:47:10.315043 19481 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:47:10.385794 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:41657
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:10.743212 19495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:10.743520 19495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:10.743718 19495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:10.751394 19495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:10.751540 19495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:10.751718 19495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:47:10.762081 19495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:41657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:10.763625 19495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:10.765079 19495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:10.779119 19500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:10.779151 19503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:10.779151 19501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:10.780507 19495 server_base.cc:1034] running on GCE node
I20260925 02:47:10.781235 19495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:10.782298 19495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:10.783686 19495 hybrid_clock.cc:648] HybridClock initialized: now 1790304430783617 us; error 45 us; skew 500 ppm
I20260925 02:47:10.784068 19495 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:10 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304430, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:10.787508 19495 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:47:10.788996 19495 webserver.cc:460] Webserver started at http://127.5.56.129:45909/ using document root <none> and password file <none>
I20260925 02:47:10.789716 19495 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:10.789816 19495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:10.790072 19495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:10.793738 19495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71"
format_stamp: "Formatted at 2026-09-25 02:47:10 on dist-test-slave-n563"
server_key: "3452aae20e2dc4e6c4deb931809e11ea"
server_key_iv: "1c4e4cc7aefb6c796bba20629e12aa9f"
server_key_version: "encryptionkey@0"
I20260925 02:47:10.794720 19495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71"
format_stamp: "Formatted at 2026-09-25 02:47:10 on dist-test-slave-n563"
server_key: "3452aae20e2dc4e6c4deb931809e11ea"
server_key_iv: "1c4e4cc7aefb6c796bba20629e12aa9f"
server_key_version: "encryptionkey@0"
I20260925 02:47:10.800329 19495 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:47:10.804019 19510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:10.805471 19495 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:47:10.805665 19495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71"
format_stamp: "Formatted at 2026-09-25 02:47:10 on dist-test-slave-n563"
server_key: "3452aae20e2dc4e6c4deb931809e11ea"
server_key_iv: "1c4e4cc7aefb6c796bba20629e12aa9f"
server_key_version: "encryptionkey@0"
I20260925 02:47:10.805905 19495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:10.843957 19495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:10.848722 19495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:10.849051 19495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:10.850090 19495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:10.851557 19495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:10.851666 19495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:10.851830 19495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:10.851889 19495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:10.886330 19495 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:41241
I20260925 02:47:10.886439 19624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:41241 every 8 connection(s)
I20260925 02:47:10.887739 19495 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:47:10.896167 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 19495
I20260925 02:47:10.896407 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:47:10.897075 5346 external_mini_cluster.cc:1428] Setting key 1e7880c82407eecceef4931baab43bc0
Sep 25 02:47:10 dist-test-slave-n563 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304430, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:10.909034 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:10.890598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:49397 (local address 127.5.56.190:41657)
0925 02:47:10.891094 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:10.891099 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:10.891971 (+ 872us) server_negotiation.cc:407] Connection header received
0925 02:47:10.893852 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:10.893860 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:10.893952 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:10.894087 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:10.895740 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:10.896585 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:10.898440 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:10.898961 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:10.899080 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:10.902973 (+ 3893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:10.902978 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:10.902981 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:10.903014 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:10.905694 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:10.906494 (+ 800us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:10.906507 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:10.906510 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:10.906631 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:10.907088 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:10.907093 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:10.907098 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:10.907388 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:10.907549 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:10.908411 (+ 862us) server_negotiation.cc:299] Negotiation successful
0925 02:47:10.908642 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":274,"threads_started":1}
I20260925 02:47:10.910228 19628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:10.891089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:49397)
0925 02:47:10.891808 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:10.891847 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:10.893574 (+ 1727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:10.894343 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:10.894351 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:10.894962 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:10.895548 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:10.895567 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:10.896753 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:10.896761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:10.898274 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:10.898287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:10.899179 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:10.899186 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:10.899451 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:10.900262 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:10.900301 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:10.902719 (+ 2418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:10.905849 (+ 3130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:10.905859 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:10.905877 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:10.906334 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:10.906754 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:10.906759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:10.906763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:10.906965 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:10.907527 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:10.907535 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:10.907986 (+ 451us) client_negotiation.cc:769] Sending connection context
0925 02:47:10.908299 (+ 313us) client_negotiation.cc:240] Negotiation successful
0925 02:47:10.908882 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":282,"threads_started":1}
I20260925 02:47:10.911901 19625 heartbeater.cc:347] Connected to a master server at 127.5.56.190:41657
I20260925 02:47:10.912240 19625 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:10.913015 19625 heartbeater.cc:511] Master 127.5.56.190:41657 requested a full tablet report, sending...
I20260925 02:47:10.915416 19433 ts_manager.cc:194] Registered new tserver with Master: 6bcbc5eece8e4daf95cf58a96ec6dd71 (127.5.56.129:41241)
I20260925 02:47:10.917618 19433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:49397
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:47:10.957437 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:41657
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:11.309506 19631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:11.309870 19631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:11.310051 19631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:11.318074 19631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:11.318246 19631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:11.318547 19631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:47:11.328913 19631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:41657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:11.330421 19631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:11.331831 19631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:11.352161 19637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:11.352404 19639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:11.352186 19636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:11.352974 19631 server_base.cc:1034] running on GCE node
I20260925 02:47:11.354027 19631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:11.355533 19631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:11.356799 19631 hybrid_clock.cc:648] HybridClock initialized: now 1790304431356718 us; error 68 us; skew 500 ppm
I20260925 02:47:11.357259 19631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:11 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:11.361670 19631 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:47:11.363515 19631 webserver.cc:460] Webserver started at http://127.5.56.130:46853/ using document root <none> and password file <none>
I20260925 02:47:11.364269 19631 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:11.364421 19631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:11.364771 19631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:11.368152 19631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "515334226e6e42c0a164b0f3ba5c646f"
format_stamp: "Formatted at 2026-09-25 02:47:11 on dist-test-slave-n563"
server_key: "675b5b6699e730a6756319ee963ee9cd"
server_key_iv: "fb7911ae9e4d6f444d75e9be5bd00557"
server_key_version: "encryptionkey@0"
I20260925 02:47:11.368916 19631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "515334226e6e42c0a164b0f3ba5c646f"
format_stamp: "Formatted at 2026-09-25 02:47:11 on dist-test-slave-n563"
server_key: "675b5b6699e730a6756319ee963ee9cd"
server_key_iv: "fb7911ae9e4d6f444d75e9be5bd00557"
server_key_version: "encryptionkey@0"
I20260925 02:47:11.374348 19631 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:47:11.377858 19646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:11.379241 19631 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:11.379419 19631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "515334226e6e42c0a164b0f3ba5c646f"
format_stamp: "Formatted at 2026-09-25 02:47:11 on dist-test-slave-n563"
server_key: "675b5b6699e730a6756319ee963ee9cd"
server_key_iv: "fb7911ae9e4d6f444d75e9be5bd00557"
server_key_version: "encryptionkey@0"
I20260925 02:47:11.379667 19631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:11.404816 19631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:11.409019 19631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:11.409430 19631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:11.410653 19631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:11.412020 19631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:11.412130 19631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:11.412284 19631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:11.412357 19631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:11.446259 19631 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:46035
I20260925 02:47:11.446352 19760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:46035 every 8 connection(s)
I20260925 02:47:11.447650 19631 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:47:11.449808 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 19631
I20260925 02:47:11.450012 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:47:11.450682 5346 external_mini_cluster.cc:1428] Setting key 4d71714cb3cd1a8c5f4933c4bc14c3e7
Sep 25 02:47:11 dist-test-slave-n563 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:11.468375 19763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:11.450354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:40077 (local address 127.5.56.190:41657)
0925 02:47:11.450886 (+ 532us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:11.450890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:11.452056 (+ 1166us) server_negotiation.cc:407] Connection header received
0925 02:47:11.454057 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:11.454062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:11.454151 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:11.454263 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:11.455810 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:11.456594 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:11.458142 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:11.458645 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:11.458777 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:11.462294 (+ 3517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:11.462299 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:11.462302 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:11.462334 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:11.465189 (+ 2855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:11.465887 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:11.465893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:11.465897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:11.466020 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:11.466538 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:11.466543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:11.466547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:11.466856 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:11.467012 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:11.467759 (+ 747us) server_negotiation.cc:299] Negotiation successful
0925 02:47:11.467986 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":258,"threads_started":1}
I20260925 02:47:11.469434 19764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:11.451057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:40077)
0925 02:47:11.451903 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:11.451941 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:11.453778 (+ 1837us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:11.454550 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:11.454558 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:11.455075 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:11.455635 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:11.455654 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:11.456768 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:11.456772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:11.457997 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:11.458009 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:11.458890 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:11.458897 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:11.459131 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:11.459845 (+ 714us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:11.459868 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:11.462041 (+ 2173us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:11.465321 (+ 3280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:11.465330 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:11.465346 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:11.465765 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:11.466151 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:11.466156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:11.466159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:11.466358 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:11.466992 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:11.467000 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:11.467358 (+ 358us) client_negotiation.cc:769] Sending connection context
0925 02:47:11.467629 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 02:47:11.468191 (+ 562us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":351,"threads_started":1}
I20260925 02:47:11.470985 19761 heartbeater.cc:347] Connected to a master server at 127.5.56.190:41657
I20260925 02:47:11.471377 19761 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:11.472375 19761 heartbeater.cc:511] Master 127.5.56.190:41657 requested a full tablet report, sending...
I20260925 02:47:11.474064 19433 ts_manager.cc:194] Registered new tserver with Master: 515334226e6e42c0a164b0f3ba5c646f (127.5.56.130:46035)
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
I20260925 02:47:11.475015 19433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:40077
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:47:11.505695 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:41657
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:11.846416 19767 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:11.846779 19767 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:11.846951 19767 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:11.855073 19767 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:11.855240 19767 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:11.855422 19767 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:47:11.865396 19767 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:41657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:11.866803 19767 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:11.868151 19767 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:11.882300 19775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:11.882249 19772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:11.882270 19773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:11.883049 19767 server_base.cc:1034] running on GCE node
I20260925 02:47:11.883811 19767 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:11.884827 19767 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:11.886056 19767 hybrid_clock.cc:648] HybridClock initialized: now 1790304431885995 us; error 37 us; skew 500 ppm
I20260925 02:47:11.886473 19767 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:11 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:11.890515 19767 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:47:11.892904 19767 webserver.cc:460] Webserver started at http://127.5.56.131:46821/ using document root <none> and password file <none>
I20260925 02:47:11.893638 19767 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:11.893801 19767 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:11.894119 19767 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:11.897272 19767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "f69cd492935f49f1bb9853c6c86bfc69"
format_stamp: "Formatted at 2026-09-25 02:47:11 on dist-test-slave-n563"
server_key: "fadde99d58df8627faeb9f94a0d6401a"
server_key_iv: "648518cdcd20b061a0ba03d169334cc5"
server_key_version: "encryptionkey@0"
I20260925 02:47:11.898077 19767 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "f69cd492935f49f1bb9853c6c86bfc69"
format_stamp: "Formatted at 2026-09-25 02:47:11 on dist-test-slave-n563"
server_key: "fadde99d58df8627faeb9f94a0d6401a"
server_key_iv: "648518cdcd20b061a0ba03d169334cc5"
server_key_version: "encryptionkey@0"
I20260925 02:47:11.903664 19767 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260925 02:47:11.907423 19782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:11.908923 19767 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:47:11.909107 19767 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "f69cd492935f49f1bb9853c6c86bfc69"
format_stamp: "Formatted at 2026-09-25 02:47:11 on dist-test-slave-n563"
server_key: "fadde99d58df8627faeb9f94a0d6401a"
server_key_iv: "648518cdcd20b061a0ba03d169334cc5"
server_key_version: "encryptionkey@0"
I20260925 02:47:11.909370 19767 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:11.920879 19625 heartbeater.cc:503] Master 127.5.56.190:41657 was elected leader, sending a full tablet report...
I20260925 02:47:11.935922 19767 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:11.940315 19767 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:11.940716 19767 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:11.941781 19767 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:11.943221 19767 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:11.943295 19767 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:11.943382 19767 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:11.943427 19767 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:11.977761 19767 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:45301
I20260925 02:47:11.977841 19895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:45301 every 8 connection(s)
I20260925 02:47:11.979172 19767 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:47:11.980731 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 19767
I20260925 02:47:11.980913 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:47:11.981375 5346 external_mini_cluster.cc:1428] Setting key d0f7c3b772f5ac0dd0c1b5be8afc6a30
Sep 25 02:47:11 dist-test-slave-n563 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304431, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:11.999892 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:11.981943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:60983 (local address 127.5.56.190:41657)
0925 02:47:11.982807 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:11.982815 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:11.983818 (+ 1003us) server_negotiation.cc:407] Connection header received
0925 02:47:11.985928 (+ 2110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:11.985933 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:11.986009 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:11.986097 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:11.987606 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:11.988371 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:11.989921 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:11.990296 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:11.990452 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:11.993892 (+ 3440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:11.993897 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:11.993900 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:11.993934 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:11.996883 (+ 2949us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:11.997777 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:11.997781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:11.997783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:11.997907 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:11.998280 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:11.998283 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:11.998285 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:11.998628 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:11.998740 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:11.999443 (+ 703us) server_negotiation.cc:299] Negotiation successful
0925 02:47:11.999615 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":593,"threads_started":1}
I20260925 02:47:12.000911 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:11.982509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:60983)
0925 02:47:11.983537 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:11.983685 (+ 148us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:11.985685 (+ 2000us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:11.986271 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:11.986279 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:11.986872 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:11.987457 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:11.987475 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:11.988497 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:11.988500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:11.989801 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:11.989813 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:11.990556 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:11.990561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:11.990800 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:11.991413 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:11.991435 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:11.993661 (+ 2226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:11.997024 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:11.997032 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:11.997046 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:11.997660 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:11.998002 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:11.998005 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:11.998007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:11.998186 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:11.998755 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:11.998761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:11.999112 (+ 351us) client_negotiation.cc:769] Sending connection context
0925 02:47:11.999345 (+ 233us) client_negotiation.cc:240] Negotiation successful
0925 02:47:11.999764 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":365,"threads_started":1}
I20260925 02:47:12.002326 19896 heartbeater.cc:347] Connected to a master server at 127.5.56.190:41657
I20260925 02:47:12.002730 19896 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:12.003505 19896 heartbeater.cc:511] Master 127.5.56.190:41657 requested a full tablet report, sending...
I20260925 02:47:12.004933 19433 ts_manager.cc:194] Registered new tserver with Master: f69cd492935f49f1bb9853c6c86bfc69 (127.5.56.131:45301)
I20260925 02:47:12.005697 19433 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:60983
I20260925 02:47:12.009786 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:47:12.027747 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.014905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38740 (local address 127.5.56.190:41657)
0925 02:47:12.015145 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:12.015153 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:12.015882 (+ 729us) server_negotiation.cc:407] Connection header received
0925 02:47:12.016062 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:12.016068 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:12.016153 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:12.016262 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:12.017821 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.018656 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.020035 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.020430 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.020526 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.022893 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:12.022898 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:12.022901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:12.022935 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:12.025392 (+ 2457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:12.026029 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:12.026033 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:12.026035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:12.026153 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:12.026534 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:12.026537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:12.026539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:12.026876 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:12.027058 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:12.027317 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 02:47:12.027491 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:47:12.031847 19433 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38740:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:47:12.038820 19433 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:12.065827 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.051481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:46035 (local address 127.0.0.1:33560)
0925 02:47:12.052657 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:12.052699 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:12.052889 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:12.055110 (+ 2221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:12.055121 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:12.055164 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:12.055760 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:12.055777 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.058673 (+ 2896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.058682 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:12.061923 (+ 3241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:12.061947 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.064642 (+ 2695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.064652 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:12.064989 (+ 337us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.065062 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 02:47:12.065189 (+ 127us) client_negotiation.cc:240] Negotiation successful
0925 02:47:12.065395 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":1347,"threads_started":1}
I20260925 02:47:12.065963 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.051686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.5.56.130:46035)
0925 02:47:12.054288 (+ 2602us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:12.054300 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:12.054336 (+ 36us) server_negotiation.cc:407] Connection header received
0925 02:47:12.054480 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:12.054488 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:12.054819 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:12.056033 (+ 1214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:12.056332 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.058474 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.062220 (+ 3746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.064352 (+ 2132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.065162 (+ 810us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.065402 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:12.065471 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 02:47:12.065651 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2400,"thread_start_us":2241,"threads_started":1}
I20260925 02:47:12.070218 19695 tablet_service.cc:1467] Processing CreateTablet for tablet ce7059d9474b4404b5dd6b4826155576 (DEFAULT_TABLE table=test-table [id=3274f10414b346899486a06891abf107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:12.071830 19695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7059d9474b4404b5dd6b4826155576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:12.073112 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.053025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.5.56.131:45301)
0925 02:47:12.062955 (+ 9930us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:12.062966 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:12.065462 (+ 2496us) server_negotiation.cc:407] Connection header received
0925 02:47:12.065571 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:12.065580 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:12.065843 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:12.066049 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:12.066985 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.068203 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.070864 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.071982 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.072131 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.072376 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:12.072628 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 02:47:12.072808 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9740,"thread_start_us":9579,"threads_started":1}
I20260925 02:47:12.073348 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.051888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:41241 (local address 127.0.0.1:52026)
0925 02:47:12.053507 (+ 1619us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:12.053530 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:12.053667 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:12.065503 (+ 11836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:12.065509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:12.065539 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:12.065930 (+ 391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:12.065947 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.067933 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.067939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:12.070511 (+ 2572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:12.070528 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.072461 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.072468 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:12.072690 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.072724 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 02:47:12.072792 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 02:47:12.072930 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1483,"mutex_wait_us":247,"thread_start_us":1344,"threads_started":1}
I20260925 02:47:12.073395 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.052434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45301 (local address 127.0.0.1:48628)
0925 02:47:12.065173 (+ 12739us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:12.065197 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:12.065322 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:12.066338 (+ 1016us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:12.066343 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:12.066368 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:12.066769 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:12.066781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.068411 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.068419 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:12.070721 (+ 2302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:12.070732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.072182 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.072188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:12.072368 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.072413 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:47:12.072588 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 02:47:12.072727 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12617,"mutex_wait_us":25,"thread_start_us":12454,"threads_started":1}
I20260925 02:47:12.074528 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.052396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.5.56.129:41241)
0925 02:47:12.064769 (+ 12373us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:12.064781 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:12.064818 (+ 37us) server_negotiation.cc:407] Connection header received
0925 02:47:12.064937 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:12.064948 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:12.065254 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:12.066123 (+ 869us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:12.066375 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.067777 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.070677 (+ 2900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.072177 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.073699 (+ 1522us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.073888 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:12.073964 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 02:47:12.074172 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12197,"thread_start_us":12058,"threads_started":1}
I20260925 02:47:12.076908 19830 tablet_service.cc:1467] Processing CreateTablet for tablet ce7059d9474b4404b5dd6b4826155576 (DEFAULT_TABLE table=test-table [id=3274f10414b346899486a06891abf107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:12.078289 19830 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7059d9474b4404b5dd6b4826155576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:12.078601 19559 tablet_service.cc:1467] Processing CreateTablet for tablet ce7059d9474b4404b5dd6b4826155576 (DEFAULT_TABLE table=test-table [id=3274f10414b346899486a06891abf107]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:12.079680 19559 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce7059d9474b4404b5dd6b4826155576. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:12.091401 19916 tablet_bootstrap.cc:492] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: Bootstrap starting.
I20260925 02:47:12.093464 19918 tablet_bootstrap.cc:492] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: Bootstrap starting.
I20260925 02:47:12.095191 19916 tablet_bootstrap.cc:654] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:12.095352 19917 tablet_bootstrap.cc:492] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: Bootstrap starting.
I20260925 02:47:12.096694 19916 log.cc:824] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:12.097131 19918 tablet_bootstrap.cc:654] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:12.098263 19918 log.cc:824] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:12.098663 19917 tablet_bootstrap.cc:654] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:12.099897 19917 log.cc:824] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:12.100108 19916 tablet_bootstrap.cc:492] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: No bootstrap required, opened a new log
I20260925 02:47:12.100404 19916 ts_tablet_manager.cc:1397] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 02:47:12.101440 19918 tablet_bootstrap.cc:492] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: No bootstrap required, opened a new log
I20260925 02:47:12.101788 19918 ts_tablet_manager.cc:1397] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 02:47:12.102825 19917 tablet_bootstrap.cc:492] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: No bootstrap required, opened a new log
I20260925 02:47:12.103116 19917 ts_tablet_manager.cc:1397] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.005s
I20260925 02:47:12.110224 19917 raft_consensus.cc:348] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:12.110771 19917 raft_consensus.cc:374] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:12.110931 19917 raft_consensus.cc:729] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515334226e6e42c0a164b0f3ba5c646f, State: Initialized, Role: FOLLOWER
I20260925 02:47:12.111598 19917 consensus_queue.cc:260] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:12.111548 19916 raft_consensus.cc:348] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:12.112018 19916 raft_consensus.cc:374] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:12.112131 19916 raft_consensus.cc:729] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bcbc5eece8e4daf95cf58a96ec6dd71, State: Initialized, Role: FOLLOWER
I20260925 02:47:12.112722 19916 consensus_queue.cc:260] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:12.113139 19918 raft_consensus.cc:348] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:12.113739 19918 raft_consensus.cc:374] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:12.113884 19918 raft_consensus.cc:729] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69cd492935f49f1bb9853c6c86bfc69, State: Initialized, Role: FOLLOWER
I20260925 02:47:12.114497 19918 consensus_queue.cc:260] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:12.114797 19916 ts_tablet_manager.cc:1428] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: Time spent starting tablet: real 0.014s user 0.012s sys 0.002s
I20260925 02:47:12.115764 19896 heartbeater.cc:503] Master 127.5.56.190:41657 was elected leader, sending a full tablet report...
I20260925 02:47:12.116423 19918 ts_tablet_manager.cc:1428] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 02:47:12.117571 19761 heartbeater.cc:503] Master 127.5.56.190:41657 was elected leader, sending a full tablet report...
I20260925 02:47:12.118186 19917 ts_tablet_manager.cc:1428] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: Time spent starting tablet: real 0.015s user 0.010s sys 0.001s
W20260925 02:47:12.140440 19626 tablet.cc:2369] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:12.194922 19925 raft_consensus.cc:482] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:12.195214 19925 raft_consensus.cc:504] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:12.196979 19925 leader_election.cc:288] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 515334226e6e42c0a164b0f3ba5c646f (127.5.56.130:46035), 6bcbc5eece8e4daf95cf58a96ec6dd71 (127.5.56.129:41241)
W20260925 02:47:12.199744 19762 tablet.cc:2369] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:12.203852 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.197272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:41241 (local address 127.5.56.131:56489)
0925 02:47:12.197604 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:12.197630 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:12.197789 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:12.198242 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:12.198246 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:12.198271 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:12.198512 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:12.198542 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.199785 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.199793 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:12.201862 (+ 2069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:12.201877 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.203300 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.203304 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:12.203439 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.203464 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:47:12.203542 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:47:12.203638 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":1}
I20260925 02:47:12.203997 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.197614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:56489 (local address 127.5.56.129:41241)
0925 02:47:12.197799 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:12.197806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:12.197827 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:47:12.197999 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:12.198006 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:12.198096 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:12.198203 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:12.198677 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.199656 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.202024 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.203157 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.203325 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.203406 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:12.203590 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 02:47:12.203734 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 02:47:12.204929 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.197365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:48299 (local address 127.5.56.130:46035)
0925 02:47:12.197604 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:12.197611 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:12.198364 (+ 753us) server_negotiation.cc:407] Connection header received
0925 02:47:12.198609 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:12.198617 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:12.198699 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:12.198766 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:12.199343 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.200426 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.202635 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.203846 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.203995 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.204078 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:12.204549 (+ 471us) server_negotiation.cc:299] Negotiation successful
0925 02:47:12.204680 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:47:12.205036 19926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.197360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:46035 (local address 127.5.56.131:48299)
0925 02:47:12.198250 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:12.198268 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:12.198470 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:12.198820 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:12.198824 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:12.198849 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:12.199189 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:12.199200 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.200569 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.200575 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:12.202489 (+ 1914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:12.202500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.204224 (+ 1724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.204230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:12.204384 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.204406 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:47:12.204512 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:47:12.204638 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"mutex_wait_us":25,"spinlock_wait_cycles":1920,"thread_start_us":542,"threads_started":1}
I20260925 02:47:12.205077 19579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7059d9474b4404b5dd6b4826155576" candidate_uuid: "f69cd492935f49f1bb9853c6c86bfc69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" is_pre_election: true
I20260925 02:47:12.205710 19579 raft_consensus.cc:2454] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f69cd492935f49f1bb9853c6c86bfc69 in term 0.
I20260925 02:47:12.206200 19715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7059d9474b4404b5dd6b4826155576" candidate_uuid: "f69cd492935f49f1bb9853c6c86bfc69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515334226e6e42c0a164b0f3ba5c646f" is_pre_election: true
I20260925 02:47:12.206549 19784 leader_election.cc:302] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bcbc5eece8e4daf95cf58a96ec6dd71, f69cd492935f49f1bb9853c6c86bfc69; no voters:
I20260925 02:47:12.206777 19715 raft_consensus.cc:2454] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f69cd492935f49f1bb9853c6c86bfc69 in term 0.
I20260925 02:47:12.206988 19925 raft_consensus.cc:2781] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:12.207134 19925 raft_consensus.cc:482] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:12.207271 19925 raft_consensus.cc:3037] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:12.209677 19925 raft_consensus.cc:504] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:12.210327 19925 leader_election.cc:288] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [CANDIDATE]: Term 1 election: Requested vote from peers 515334226e6e42c0a164b0f3ba5c646f (127.5.56.130:46035), 6bcbc5eece8e4daf95cf58a96ec6dd71 (127.5.56.129:41241)
I20260925 02:47:12.210736 19715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7059d9474b4404b5dd6b4826155576" candidate_uuid: "f69cd492935f49f1bb9853c6c86bfc69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515334226e6e42c0a164b0f3ba5c646f"
I20260925 02:47:12.210819 19579 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce7059d9474b4404b5dd6b4826155576" candidate_uuid: "f69cd492935f49f1bb9853c6c86bfc69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71"
I20260925 02:47:12.211004 19715 raft_consensus.cc:3037] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:12.211021 19579 raft_consensus.cc:3037] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:12.213336 19579 raft_consensus.cc:2454] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f69cd492935f49f1bb9853c6c86bfc69 in term 1.
I20260925 02:47:12.213645 19715 raft_consensus.cc:2454] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f69cd492935f49f1bb9853c6c86bfc69 in term 1.
I20260925 02:47:12.213814 19784 leader_election.cc:302] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bcbc5eece8e4daf95cf58a96ec6dd71, f69cd492935f49f1bb9853c6c86bfc69; no voters:
I20260925 02:47:12.214190 19925 raft_consensus.cc:2781] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:12.214680 19925 raft_consensus.cc:686] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 1 LEADER]: Becoming Leader. State: Replica: f69cd492935f49f1bb9853c6c86bfc69, State: Running, Role: LEADER
I20260925 02:47:12.215257 19925 consensus_queue.cc:237] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:12.220391 19431 catalog_manager.cc:5495] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 reported cstate change: term changed from 0 to 1, leader changed from <none> to f69cd492935f49f1bb9853c6c86bfc69 (127.5.56.131). New cstate: current_term: 1 leader_uuid: "f69cd492935f49f1bb9853c6c86bfc69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } health_report { overall_health: HEALTHY } } }
W20260925 02:47:12.230758 19897 tablet.cc:2369] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:12.256028 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.250352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48640 (local address 127.5.56.131:45301)
0925 02:47:12.250578 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:12.250585 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:12.250627 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:47:12.250915 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:12.250919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:12.250997 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:12.251155 (+ 158us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:12.251539 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.252288 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.253783 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.254189 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.254333 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.254912 (+ 579us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:12.255430 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:12.255597 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:12.255676 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 02:47:12.255761 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 02:47:12.264283 19579 raft_consensus.cc:1261] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 1 FOLLOWER]: Refusing update from remote peer f69cd492935f49f1bb9853c6c86bfc69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:12.264279 19715 raft_consensus.cc:1261] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f [term 1 FOLLOWER]: Refusing update from remote peer f69cd492935f49f1bb9853c6c86bfc69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:12.265322 19927 consensus_queue.cc:1035] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:12.265661 19925 consensus_queue.cc:1035] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:12.278980 19931 mvcc.cc:205] Tried to move back new op lower bound from 7333086954544263168 to 7333086954357964800. Current Snapshot: MvccSnapshot[applied={T|T < 7333086954544263168}]
I20260925 02:47:12.299667 19431 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:12.300091 19431 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38740:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:12.305485 19431 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Sending DeleteTablet for 3 replicas of tablet ce7059d9474b4404b5dd6b4826155576
I20260925 02:47:12.306882 19830 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7059d9474b4404b5dd6b4826155576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:12 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:48628
I20260925 02:47:12.307067 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 19495
I20260925 02:47:12.307523 19695 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7059d9474b4404b5dd6b4826155576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:12 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:33560
I20260925 02:47:12.309772 19559 tablet_service.cc:1514] Processing DeleteTablet for tablet ce7059d9474b4404b5dd6b4826155576 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:12 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:52026
I20260925 02:47:12.310683 19944 tablet_replica.cc:330] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: stopping tablet replica
I20260925 02:47:12.311141 19944 raft_consensus.cc:2229] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:12.311600 19944 raft_consensus.cc:2258] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:12.313349 19944 ts_tablet_manager.cc:1905] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:12.316202 19942 tablet_replica.cc:330] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: stopping tablet replica
I20260925 02:47:12.316772 19942 raft_consensus.cc:2229] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:12.317283 19942 raft_consensus.cc:2258] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:12.318691 19944 ts_tablet_manager.cc:1918] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:12.318918 19944 log.cc:1196] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/ce7059d9474b4404b5dd6b4826155576
I20260925 02:47:12.319564 19941 tablet_replica.cc:330] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: stopping tablet replica
I20260925 02:47:12.319577 19944 ts_tablet_manager.cc:1939] T ce7059d9474b4404b5dd6b4826155576 P 6bcbc5eece8e4daf95cf58a96ec6dd71: Deleting consensus metadata
I20260925 02:47:12.320199 19941 raft_consensus.cc:2229] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:47:12.320935 19941 raft_consensus.cc:2258] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:12.321686 19418 catalog_manager.cc:4841] TS 6bcbc5eece8e4daf95cf58a96ec6dd71 (127.5.56.129:41241): tablet ce7059d9474b4404b5dd6b4826155576 (table test-table [id=3274f10414b346899486a06891abf107]) successfully deleted
I20260925 02:47:12.323082 19941 ts_tablet_manager.cc:1905] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:12.324126 19942 ts_tablet_manager.cc:1905] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:12.330560 19941 ts_tablet_manager.cc:1918] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:12.330802 19941 log.cc:1196] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/ce7059d9474b4404b5dd6b4826155576
I20260925 02:47:12.331425 19941 ts_tablet_manager.cc:1939] T ce7059d9474b4404b5dd6b4826155576 P f69cd492935f49f1bb9853c6c86bfc69: Deleting consensus metadata
Sep 25 02:47:12 dist-test-slave-n563 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:47:12.331619 19942 ts_tablet_manager.cc:1918] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:12.331822 19942 log.cc:1196] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/ce7059d9474b4404b5dd6b4826155576
I20260925 02:47:12.332398 19942 ts_tablet_manager.cc:1939] T ce7059d9474b4404b5dd6b4826155576 P 515334226e6e42c0a164b0f3ba5c646f: Deleting consensus metadata
I20260925 02:47:12.334151 19420 catalog_manager.cc:4841] TS f69cd492935f49f1bb9853c6c86bfc69 (127.5.56.131:45301): tablet ce7059d9474b4404b5dd6b4826155576 (table test-table [id=3274f10414b346899486a06891abf107]) successfully deleted
I20260925 02:47:12.334404 19419 catalog_manager.cc:4841] TS 515334226e6e42c0a164b0f3ba5c646f (127.5.56.130:46035): tablet ce7059d9474b4404b5dd6b4826155576 (table test-table [id=3274f10414b346899486a06891abf107]) successfully deleted
I20260925 02:47:12.340417 19913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.308962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52034 (local address 127.5.56.129:41241)
0925 02:47:12.309159 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:12.309165 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:12.319169 (+ 10004us) server_negotiation.cc:407] Connection header received
0925 02:47:12.319400 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:12.319406 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:12.319496 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:12.319626 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:12.322171 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.323095 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.326237 (+ 3142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.326906 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.327004 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.333139 (+ 6135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:12.333161 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:12.333179 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:12.333232 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:12.337282 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:12.338057 (+ 775us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:12.338068 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:12.338073 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:12.338255 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:12.338685 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:12.338692 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:12.338695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:12.339189 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:12.339518 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:12.339764 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 02:47:12.340006 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:47:12.341179 19620 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:12.433833 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 19495
I20260925 02:47:12.455082 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 19631
Sep 25 02:47:12 dist-test-slave-n563 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:47:12.471290 19911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.455906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33564 (local address 127.5.56.130:46035)
0925 02:47:12.456090 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:12.456097 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:12.456244 (+ 147us) server_negotiation.cc:407] Connection header received
0925 02:47:12.456448 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:12.456454 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:12.456526 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:12.456653 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:12.457939 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.458787 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.460099 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.460527 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.460663 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.465085 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:12.465098 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:12.465113 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:12.465154 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:12.468210 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:12.468976 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:12.468983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:12.468986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:12.469107 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:12.469525 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:12.469532 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:12.469540 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:12.469960 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:12.470272 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:12.470619 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 02:47:12.470867 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:47:12.472144 19756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:12.564289 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 19631
I20260925 02:47:12.582974 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 19767
Sep 25 02:47:12 dist-test-slave-n563 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:47:12.601956 19915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:12.583921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48642 (local address 127.5.56.131:45301)
0925 02:47:12.584136 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:12.584143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:12.584167 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:47:12.584248 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:12.584253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:12.584337 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:12.584457 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:12.585827 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.587332 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.588786 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:12.589500 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:12.589633 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:12.594225 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:12.594243 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:12.594265 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:12.594324 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:12.598740 (+ 4416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:12.599541 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:12.599552 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:12.599555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:12.599767 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:12.600167 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:12.600172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:12.600175 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:12.600697 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:12.601022 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:12.601262 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 02:47:12.601514 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260925 02:47:12.603003 19891 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:12.731817 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 19767
I20260925 02:47:12.752871 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 19401
I20260925 02:47:12.754071 19464 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:12.868996 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 19401
I20260925 02:47:12.891800 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41657
--webserver_interface=127.5.56.190
--webserver_port=37409
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:41657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:13.238276 19949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:13.238735 19949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:13.238906 19949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:13.246438 19949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:13.246615 19949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:13.246719 19949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:13.246809 19949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:13.246893 19949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:13.257496 19949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:41657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=37409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:13.259105 19949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:13.260465 19949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:13.285257 19955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:13.285269 19958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:13.285215 19956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:13.288242 19949 server_base.cc:1034] running on GCE node
I20260925 02:47:13.289204 19949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:13.291071 19949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:13.292302 19949 hybrid_clock.cc:648] HybridClock initialized: now 1790304433292237 us; error 48 us; skew 500 ppm
I20260925 02:47:13.292757 19949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:13 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304433, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:13.296504 19949 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:47:13.298333 19949 webserver.cc:460] Webserver started at http://127.5.56.190:37409/ using document root <none> and password file <none>
I20260925 02:47:13.299209 19949 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:13.299381 19949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:13.306013 19949 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260925 02:47:13.309422 19965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:13.311036 19949 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:13.311255 19949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "f67d111f1c2845a4a117f2cb83d5eb64"
format_stamp: "Formatted at 2026-09-25 02:47:10 on dist-test-slave-n563"
server_key: "5df4bf6bb4374b102932869e59529785"
server_key_iv: "8d508034d72501bb7097ef75d34ee2c5"
server_key_version: "encryptionkey@0"
I20260925 02:47:13.312515 19949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:13.336342 19949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:13.340853 19949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:13.341172 19949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:13.361055 19949 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:41657
I20260925 02:47:13.361063 20016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:41657 every 8 connection(s)
I20260925 02:47:13.362748 19949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:47:13.368988 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 19949
I20260925 02:47:13.370045 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:41241
--local_ip_for_outbound_sockets=127.5.56.129
--tserver_master_addrs=127.5.56.190:41657
--webserver_port=45909
--webserver_interface=127.5.56.129
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:47:13.370877 20017 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:47:13.376434 20017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Bootstrap starting.
I20260925 02:47:13.397662 20017 log.cc:824] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:13.410815 20017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:47:13.411448 20017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Bootstrap complete.
I20260925 02:47:13.421263 20017 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:13.422091 20017 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67d111f1c2845a4a117f2cb83d5eb64, State: Initialized, Role: FOLLOWER
I20260925 02:47:13.422806 20017 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:13.423049 20017 raft_consensus.cc:388] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:47:13.423166 20017 raft_consensus.cc:482] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:47:13.423369 20017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:47:13.427464 20017 raft_consensus.cc:504] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:13.428030 20017 leader_election.cc:302] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f67d111f1c2845a4a117f2cb83d5eb64; no voters:
I20260925 02:47:13.428709 20017 leader_election.cc:288] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:47:13.428853 20021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:47:13.430145 20021 raft_consensus.cc:686] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 2 LEADER]: Becoming Leader. State: Replica: f67d111f1c2845a4a117f2cb83d5eb64, State: Running, Role: LEADER
I20260925 02:47:13.430765 20021 consensus_queue.cc:237] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:13.431227 20017 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:13.433444 20022 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } } }
I20260925 02:47:13.433881 20022 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:13.437768 20023 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f67d111f1c2845a4a117f2cb83d5eb64. Latest consensus state: current_term: 2 leader_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } } }
I20260925 02:47:13.438148 20023 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:13.440002 20027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:47:13.450486 20027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:47:13.452351 20027 catalog_manager.cc:1259] Loaded cluster ID: d06b4eec85da4c08899127f75c6d45d7
I20260925 02:47:13.452485 20027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:47:13.456661 20027 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 02:47:13.457371 20038 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 02:47:13.468904 20027 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 02:47:13.476284 20027 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:47:13.477852 20027 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Loaded TSK: 0
I20260925 02:47:13.479159 20027 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:47:13.748420 20019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:13.748700 20019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:13.748847 20019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:13.756488 20019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:13.756603 20019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:13.756774 20019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:47:13.767283 20019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:41241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=45909
--tserver_master_addrs=127.5.56.190:41657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:13.768676 20019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:13.770092 20019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:13.785857 20047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:13.785887 20045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:13.786739 20044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:13.787407 20019 server_base.cc:1034] running on GCE node
I20260925 02:47:13.788234 20019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:13.789601 20019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:13.790859 20019 hybrid_clock.cc:648] HybridClock initialized: now 1790304433790788 us; error 57 us; skew 500 ppm
I20260925 02:47:13.791260 20019 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:13 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304433, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:13.795199 20019 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:47:13.796979 20019 webserver.cc:460] Webserver started at http://127.5.56.129:45909/ using document root <none> and password file <none>
I20260925 02:47:13.797844 20019 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:13.797988 20019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:13.805235 20019 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260925 02:47:13.808728 20054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:13.810580 20019 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:47:13.810763 20019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71"
format_stamp: "Formatted at 2026-09-25 02:47:10 on dist-test-slave-n563"
server_key: "3452aae20e2dc4e6c4deb931809e11ea"
server_key_iv: "1c4e4cc7aefb6c796bba20629e12aa9f"
server_key_version: "encryptionkey@0"
I20260925 02:47:13.812175 20019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:13.837320 20019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:13.842978 20019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:13.843281 20019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:13.844511 20019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:13.846172 20019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:13.846278 20019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:13.846410 20019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:13.846498 20019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260925 02:47:13.881374 20019 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:41241
I20260925 02:47:13.881428 20169 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:41241 every 8 connection(s)
I20260925 02:47:13.883003 20019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:47:13.889313 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 20019
I20260925 02:47:13.890192 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:46035
--local_ip_for_outbound_sockets=127.5.56.130
--tserver_master_addrs=127.5.56.190:41657
--webserver_port=46853
--webserver_interface=127.5.56.130
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:47:13 dist-test-slave-n563 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304433, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:13.909703 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:13.885984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:49853 (local address 127.5.56.190:41657)
0925 02:47:13.886736 (+ 752us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:13.886752 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:13.887231 (+ 479us) server_negotiation.cc:407] Connection header received
0925 02:47:13.889344 (+ 2113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:13.889366 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:13.889868 (+ 502us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:13.890289 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:13.891917 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:13.893931 (+ 2014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:13.895598 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:13.896710 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:13.896964 (+ 254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:13.901666 (+ 4702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:13.901675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:13.901693 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:13.901741 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:13.904880 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:13.905941 (+ 1061us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:13.905947 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:13.905953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:13.906124 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:13.906680 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:13.906684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:13.906686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:13.907195 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:13.907352 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:13.908350 (+ 998us) server_negotiation.cc:299] Negotiation successful
0925 02:47:13.908702 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":292,"threads_started":1}
I20260925 02:47:13.910161 20173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:13.886227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:49853)
0925 02:47:13.887041 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:13.887095 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:13.888722 (+ 1627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:13.890558 (+ 1836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:13.890570 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:13.891097 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:13.891648 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:13.891666 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:13.894109 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:13.894113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:13.895435 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:13.895448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:13.896995 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:13.897000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:13.897189 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:13.897955 (+ 766us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:13.898001 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:13.901337 (+ 3336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:13.905056 (+ 3719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:13.905070 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:13.905091 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:13.905786 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:13.906251 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:13.906258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:13.906265 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:13.906553 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:13.907344 (+ 791us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:13.907355 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:13.907834 (+ 479us) client_negotiation.cc:769] Sending connection context
0925 02:47:13.908162 (+ 328us) client_negotiation.cc:240] Negotiation successful
0925 02:47:13.908776 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":328,"threads_started":1}
I20260925 02:47:13.913587 20170 heartbeater.cc:347] Connected to a master server at 127.5.56.190:41657
I20260925 02:47:13.913913 20170 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:13.914746 20170 heartbeater.cc:511] Master 127.5.56.190:41657 requested a full tablet report, sending...
I20260925 02:47:13.917541 19982 ts_manager.cc:194] Registered new tserver with Master: 6bcbc5eece8e4daf95cf58a96ec6dd71 (127.5.56.129:41241)
I20260925 02:47:13.919730 19982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:49853
W20260925 02:47:14.240765 20174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:14.241078 20174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:14.241175 20174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:14.248716 20174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:14.248847 20174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:14.248979 20174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:47:14.259196 20174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:46035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=46853
--tserver_master_addrs=127.5.56.190:41657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:14.260830 20174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:14.262179 20174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:14.276110 20179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:14.276031 20180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:14.276031 20182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:14.277150 20174 server_base.cc:1034] running on GCE node
I20260925 02:47:14.277887 20174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:14.278939 20174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:14.280138 20174 hybrid_clock.cc:648] HybridClock initialized: now 1790304434280074 us; error 56 us; skew 500 ppm
I20260925 02:47:14.280473 20174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:14 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304434, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:14.283663 20174 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:47:14.285279 20174 webserver.cc:460] Webserver started at http://127.5.56.130:46853/ using document root <none> and password file <none>
I20260925 02:47:14.286027 20174 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:14.286185 20174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:14.291961 20174 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260925 02:47:14.294782 20189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:14.296165 20174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:47:14.296357 20174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "515334226e6e42c0a164b0f3ba5c646f"
format_stamp: "Formatted at 2026-09-25 02:47:11 on dist-test-slave-n563"
server_key: "675b5b6699e730a6756319ee963ee9cd"
server_key_iv: "fb7911ae9e4d6f444d75e9be5bd00557"
server_key_version: "encryptionkey@0"
I20260925 02:47:14.297551 20174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:14.340076 20174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:14.344211 20174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:14.344501 20174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:14.345592 20174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:14.346948 20174 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:14.347081 20174 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:14.347213 20174 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:14.347302 20174 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:14.383297 20174 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:46035
I20260925 02:47:14.383386 20302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:46035 every 8 connection(s)
I20260925 02:47:14.384817 20174 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:47:14.389990 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 20174
I20260925 02:47:14.390671 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:45301
--local_ip_for_outbound_sockets=127.5.56.131
--tserver_master_addrs=127.5.56.190:41657
--webserver_port=46821
--webserver_interface=127.5.56.131
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:47:14 dist-test-slave-n563 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304434, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:14.406523 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.387442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:59643 (local address 127.5.56.190:41657)
0925 02:47:14.387668 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:14.387675 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:14.388879 (+ 1204us) server_negotiation.cc:407] Connection header received
0925 02:47:14.390853 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:14.390863 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:14.390995 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:14.391112 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:14.392657 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.393757 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.395454 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.396197 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.396352 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.400167 (+ 3815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:14.400172 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:14.400175 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:14.400216 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:14.403192 (+ 2976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:14.404001 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:14.404006 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:14.404010 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:14.404156 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:14.404612 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:14.404617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:14.404620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:14.404923 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:14.405119 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:14.405878 (+ 759us) server_negotiation.cc:299] Negotiation successful
0925 02:47:14.406142 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:47:14.407639 20305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.387973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:59643)
0925 02:47:14.388693 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:14.388736 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:14.390557 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:14.391345 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:14.391354 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:14.391875 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:14.392463 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:14.392481 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.393932 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.393936 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:14.395285 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:14.395299 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.396420 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.396424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:14.396632 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.397378 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:14.397401 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:14.399911 (+ 2510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:14.403367 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:14.403376 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:14.403394 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:14.403860 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:14.404274 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:14.404281 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:14.404286 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:14.404488 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:14.405087 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:14.405096 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:14.405484 (+ 388us) client_negotiation.cc:769] Sending connection context
0925 02:47:14.405764 (+ 280us) client_negotiation.cc:240] Negotiation successful
0925 02:47:14.406354 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":281,"threads_started":1}
I20260925 02:47:14.409235 20303 heartbeater.cc:347] Connected to a master server at 127.5.56.190:41657
I20260925 02:47:14.409648 20303 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:14.410756 20303 heartbeater.cc:511] Master 127.5.56.190:41657 requested a full tablet report, sending...
I20260925 02:47:14.412736 19982 ts_manager.cc:194] Registered new tserver with Master: 515334226e6e42c0a164b0f3ba5c646f (127.5.56.130:46035)
I20260925 02:47:14.414129 19982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:59643
W20260925 02:47:14.736918 20306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:14.737221 20306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:14.737327 20306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:14.744840 20306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:14.744969 20306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:14.745088 20306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:47:14.755182 20306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:45301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=46821
--tserver_master_addrs=127.5.56.190:41657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:14.756492 20306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:14.757891 20306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:14.771667 20311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:14.771695 20314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:14.771667 20312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:14.772871 20306 server_base.cc:1034] running on GCE node
I20260925 02:47:14.773505 20306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:14.774505 20306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:14.775725 20306 hybrid_clock.cc:648] HybridClock initialized: now 1790304434775665 us; error 56 us; skew 500 ppm
I20260925 02:47:14.776054 20306 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:14 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304434, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:14.779206 20306 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:47:14.780781 20306 webserver.cc:460] Webserver started at http://127.5.56.131:46821/ using document root <none> and password file <none>
I20260925 02:47:14.781525 20306 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:14.781682 20306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:14.787443 20306 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260925 02:47:14.790120 20321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:14.791544 20306 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:47:14.791730 20306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "f69cd492935f49f1bb9853c6c86bfc69"
format_stamp: "Formatted at 2026-09-25 02:47:11 on dist-test-slave-n563"
server_key: "fadde99d58df8627faeb9f94a0d6401a"
server_key_iv: "648518cdcd20b061a0ba03d169334cc5"
server_key_version: "encryptionkey@0"
I20260925 02:47:14.792748 20306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:14.804064 20306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:14.808079 20306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:14.808379 20306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:14.809445 20306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:14.810875 20306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:14.811043 20306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:14.811175 20306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:14.811270 20306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:14.845201 20306 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:45301
I20260925 02:47:14.845293 20434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:45301 every 8 connection(s)
I20260925 02:47:14.846793 20306 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:47:14.848644 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 20306
Sep 25 02:47:14 dist-test-slave-n563 krb5kdc[19388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304434, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:14.865142 20437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.849626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38746 (local address 127.5.56.190:41657)
0925 02:47:14.850120 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:14.850128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:14.850504 (+ 376us) server_negotiation.cc:407] Connection header received
0925 02:47:14.850760 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:14.850765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:14.850861 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:14.850979 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:14.852708 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.853865 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.855415 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.855844 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.855933 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.859160 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:14.859166 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:14.859170 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:14.859209 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:14.862208 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:14.862922 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:14.862927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:14.862936 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:14.863078 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:14.863523 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:14.863527 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:14.863529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:14.863824 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:14.864118 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:14.864465 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 02:47:14.864783 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":251,"threads_started":1}
I20260925 02:47:14.869120 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.849483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:40805 (local address 127.5.56.190:41657)
0925 02:47:14.849682 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:14.849689 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:14.851248 (+ 1559us) server_negotiation.cc:407] Connection header received
0925 02:47:14.853368 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:14.853371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:14.853430 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:14.853501 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:14.855194 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.856055 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.857795 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.858628 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.858838 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.862874 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:14.862879 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:14.862882 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:14.862912 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:14.866024 (+ 3112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:14.866835 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:14.866839 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:14.866841 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:14.866948 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:14.867421 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:14.867425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:14.867426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:14.867753 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:14.867902 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:14.868601 (+ 699us) server_negotiation.cc:299] Negotiation successful
0925 02:47:14.868798 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":9,"server-negotiator.queue_time_us":56}
I20260925 02:47:14.870223 20439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.850164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:40805)
0925 02:47:14.851086 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:14.851140 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:14.853086 (+ 1946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:14.853731 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:14.853741 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:14.854376 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:14.855026 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:14.855043 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.856209 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.856212 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:14.857609 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:14.857621 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.858889 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.858893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:14.859160 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.859868 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:14.859901 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:14.862649 (+ 2748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:14.866159 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:14.866168 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:14.866187 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:14.866711 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:14.867053 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:14.867061 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:14.867066 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:14.867313 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:14.867889 (+ 576us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:14.867897 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:14.868233 (+ 336us) client_negotiation.cc:769] Sending connection context
0925 02:47:14.868494 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 02:47:14.869051 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":370,"threads_started":1}
I20260925 02:47:14.871549 20435 heartbeater.cc:347] Connected to a master server at 127.5.56.190:41657
I20260925 02:47:14.871845 20435 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:14.872530 20435 heartbeater.cc:511] Master 127.5.56.190:41657 requested a full tablet report, sending...
I20260925 02:47:14.873884 19982 ts_manager.cc:194] Registered new tserver with Master: f69cd492935f49f1bb9853c6c86bfc69 (127.5.56.131:45301)
I20260925 02:47:14.874759 19982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:40805
I20260925 02:47:14.880119 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:47:14.887609 20172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.881110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.5.56.190:41657)
0925 02:47:14.881300 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:14.881308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:14.881904 (+ 596us) server_negotiation.cc:407] Connection header received
0925 02:47:14.882144 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:14.882148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:14.882280 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:14.882487 (+ 207us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:14.882929 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.883630 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.885282 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.885727 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.885864 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.886546 (+ 682us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:14.887016 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:14.887170 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:14.887269 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 02:47:14.887355 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:47:14.888561 19982 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:47:14.895859 19982 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:14.923075 20170 heartbeater.cc:503] Master 127.5.56.190:41657 was elected leader, sending a full tablet report...
I20260925 02:47:14.928965 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.908779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.5.56.130:46035)
0925 02:47:14.919479 (+ 10700us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:14.919492 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:14.919535 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:47:14.919638 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:14.919646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:14.920114 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:14.920364 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:14.921558 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.923190 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.926127 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.927673 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.927820 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.928023 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:14.928474 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 02:47:14.928645 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10262,"thread_start_us":10208,"threads_started":1}
I20260925 02:47:14.928967 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.912214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.5.56.131:45301)
0925 02:47:14.919866 (+ 7652us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:14.919874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:14.921470 (+ 1596us) server_negotiation.cc:407] Connection header received
0925 02:47:14.921631 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:14.921639 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:14.921843 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:14.922026 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:14.922819 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.924560 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.926681 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.927607 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.927765 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.928046 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:14.928422 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 02:47:14.928598 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7474,"thread_start_us":7292,"threads_started":1}
I20260925 02:47:14.929024 20445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.911461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45301 (local address 127.0.0.1:48654)
0925 02:47:14.921321 (+ 9860us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:14.921341 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:14.921479 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:14.922146 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:14.922151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:14.922186 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:14.922623 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:14.922633 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.924710 (+ 2077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.924716 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:14.926526 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:14.926539 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.927851 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.927858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:14.928144 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.928223 (+ 79us) client_negotiation.cc:769] Sending connection context
0925 02:47:14.928362 (+ 139us) client_negotiation.cc:240] Negotiation successful
0925 02:47:14.928587 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9746,"mutex_wait_us":54,"thread_start_us":6496,"threads_started":1}
I20260925 02:47:14.929703 20446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.908030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:46035 (local address 127.0.0.1:33566)
0925 02:47:14.918556 (+ 10526us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:14.918575 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:14.919095 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:14.920387 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:14.920401 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:14.920427 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:14.921096 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:14.921117 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.923530 (+ 2413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.923535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:14.925964 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:14.925976 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.928149 (+ 2173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.928154 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:14.928291 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.928314 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:47:14.928418 (+ 104us) client_negotiation.cc:240] Negotiation successful
0925 02:47:14.928850 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10337,"mutex_wait_us":422,"spinlock_wait_cycles":796032,"thread_start_us":9977,"threads_started":1}
I20260925 02:47:14.931861 20449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.907850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:41241 (local address 127.0.0.1:52036)
0925 02:47:14.918513 (+ 10663us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:14.918567 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:14.918785 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:14.921832 (+ 3047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:14.921837 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:14.921866 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:14.922199 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:14.922211 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.924762 (+ 2551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.924769 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:14.928623 (+ 3854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:14.928636 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.930909 (+ 2273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.930916 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:14.931099 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.931131 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:47:14.931271 (+ 140us) client_negotiation.cc:240] Negotiation successful
0925 02:47:14.931451 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10284,"mutex_wait_us":470,"spinlock_wait_cycles":5376,"thread_start_us":13174,"threads_started":1}
I20260925 02:47:14.931861 20447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:14.908130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52036 (local address 127.5.56.129:41241)
0925 02:47:14.921229 (+ 13099us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:14.921237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:14.921267 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:47:14.921364 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:14.921370 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:14.921632 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:14.921843 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:14.922443 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.924061 (+ 1618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.929139 (+ 5078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:14.930682 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:14.930831 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:14.931064 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:14.931317 (+ 253us) server_negotiation.cc:299] Negotiation successful
0925 02:47:14.931522 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12900,"thread_start_us":12996,"threads_started":1}
I20260925 02:47:14.933476 20369 tablet_service.cc:1467] Processing CreateTablet for tablet 01c0370656d1463ea7e832728c03a69b (DEFAULT_TABLE table=test-table [id=66bcac8545274367969857b20c66de94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:14.933764 20237 tablet_service.cc:1467] Processing CreateTablet for tablet 01c0370656d1463ea7e832728c03a69b (DEFAULT_TABLE table=test-table [id=66bcac8545274367969857b20c66de94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:14.935096 20369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c0370656d1463ea7e832728c03a69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:14.935096 20237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c0370656d1463ea7e832728c03a69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:14.934975 20104 tablet_service.cc:1467] Processing CreateTablet for tablet 01c0370656d1463ea7e832728c03a69b (DEFAULT_TABLE table=test-table [id=66bcac8545274367969857b20c66de94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:14.936168 20104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01c0370656d1463ea7e832728c03a69b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:14.951397 20453 tablet_bootstrap.cc:492] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: Bootstrap starting.
I20260925 02:47:14.952258 20451 tablet_bootstrap.cc:492] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: Bootstrap starting.
I20260925 02:47:14.953706 20452 tablet_bootstrap.cc:492] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: Bootstrap starting.
I20260925 02:47:14.956141 20451 tablet_bootstrap.cc:654] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:14.957187 20451 log.cc:824] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:14.957379 20453 tablet_bootstrap.cc:654] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:14.957758 20452 tablet_bootstrap.cc:654] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:14.958695 20453 log.cc:824] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:14.959018 20452 log.cc:824] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:14.960569 20451 tablet_bootstrap.cc:492] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: No bootstrap required, opened a new log
I20260925 02:47:14.960865 20451 ts_tablet_manager.cc:1397] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 02:47:14.961587 20453 tablet_bootstrap.cc:492] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: No bootstrap required, opened a new log
I20260925 02:47:14.961843 20453 ts_tablet_manager.cc:1397] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20260925 02:47:14.962028 20452 tablet_bootstrap.cc:492] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: No bootstrap required, opened a new log
I20260925 02:47:14.962291 20452 ts_tablet_manager.cc:1397] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 02:47:14.969969 20451 raft_consensus.cc:348] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:14.970402 20451 raft_consensus.cc:374] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:14.970534 20451 raft_consensus.cc:729] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bcbc5eece8e4daf95cf58a96ec6dd71, State: Initialized, Role: FOLLOWER
I20260925 02:47:14.971201 20451 consensus_queue.cc:260] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:14.971419 20453 raft_consensus.cc:348] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:14.971920 20453 raft_consensus.cc:374] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:14.972050 20453 raft_consensus.cc:729] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f69cd492935f49f1bb9853c6c86bfc69, State: Initialized, Role: FOLLOWER
I20260925 02:47:14.972783 20453 consensus_queue.cc:260] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:14.973493 20452 raft_consensus.cc:348] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:14.974082 20452 raft_consensus.cc:374] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:14.974112 20451 ts_tablet_manager.cc:1428] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: Time spent starting tablet: real 0.013s user 0.009s sys 0.005s
I20260925 02:47:14.974270 20452 raft_consensus.cc:729] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515334226e6e42c0a164b0f3ba5c646f, State: Initialized, Role: FOLLOWER
I20260925 02:47:14.975078 20435 heartbeater.cc:503] Master 127.5.56.190:41657 was elected leader, sending a full tablet report...
I20260925 02:47:14.975023 20452 consensus_queue.cc:260] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:14.975775 20453 ts_tablet_manager.cc:1428] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260925 02:47:14.977415 20303 heartbeater.cc:503] Master 127.5.56.190:41657 was elected leader, sending a full tablet report...
I20260925 02:47:14.978873 20452 ts_tablet_manager.cc:1428] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
W20260925 02:47:15.098508 20436 tablet.cc:2369] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:47:15.135560 20171 tablet.cc:2369] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:47:15.136790 20304 tablet.cc:2369] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:15.170995 20457 raft_consensus.cc:482] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:15.171379 20457 raft_consensus.cc:504] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:15.173134 20457 leader_election.cc:288] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 515334226e6e42c0a164b0f3ba5c646f (127.5.56.130:46035), f69cd492935f49f1bb9853c6c86bfc69 (127.5.56.131:45301)
I20260925 02:47:15.180243 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:15.173660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:45355 (local address 127.5.56.131:45301)
0925 02:47:15.173897 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:15.173907 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:15.174883 (+ 976us) server_negotiation.cc:407] Connection header received
0925 02:47:15.175078 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:15.175085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:15.175153 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:15.175249 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:15.175918 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.176627 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.178453 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.179257 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.179441 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:15.179526 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:15.179836 (+ 310us) server_negotiation.cc:299] Negotiation successful
0925 02:47:15.179968 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:47:15.180253 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:15.173690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:54695 (local address 127.5.56.130:46035)
0925 02:47:15.173909 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:15.173917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:15.174829 (+ 912us) server_negotiation.cc:407] Connection header received
0925 02:47:15.175042 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:15.175049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:15.175115 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:15.175212 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:15.175844 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.176550 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.178450 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.179235 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.179422 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:15.179505 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:15.179836 (+ 331us) server_negotiation.cc:299] Negotiation successful
0925 02:47:15.179969 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:47:15.180416 20461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:15.173659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:46035 (local address 127.5.56.129:54695)
0925 02:47:15.174686 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:15.174702 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:15.174906 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:15.175227 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:15.175232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:15.175265 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:15.175691 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:15.175704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.176677 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.176684 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:15.178276 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:15.178289 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.179461 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.179467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:15.179651 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:15.179696 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:47:15.179810 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:47:15.179952 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":895,"mutex_wait_us":59,"thread_start_us":539,"threads_started":1}
I20260925 02:47:15.180565 20460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:15.173700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:45301 (local address 127.5.56.129:45355)
0925 02:47:15.174767 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:15.174781 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:15.174940 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:15.175267 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:15.175271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:15.175293 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:15.175726 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:15.175735 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.176788 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.176792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:15.178271 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:15.178283 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.179499 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.179505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:15.179664 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:15.179703 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:47:15.179810 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:47:15.179955 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":976,"mutex_wait_us":13,"thread_start_us":459,"threads_started":1}
I20260925 02:47:15.181840 20258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c0370656d1463ea7e832728c03a69b" candidate_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515334226e6e42c0a164b0f3ba5c646f" is_pre_election: true
I20260925 02:47:15.181845 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c0370656d1463ea7e832728c03a69b" candidate_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69cd492935f49f1bb9853c6c86bfc69" is_pre_election: true
I20260925 02:47:15.182334 20258 raft_consensus.cc:2454] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bcbc5eece8e4daf95cf58a96ec6dd71 in term 0.
I20260925 02:47:15.182332 20390 raft_consensus.cc:2454] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bcbc5eece8e4daf95cf58a96ec6dd71 in term 0.
I20260925 02:47:15.183037 20058 leader_election.cc:302] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bcbc5eece8e4daf95cf58a96ec6dd71, f69cd492935f49f1bb9853c6c86bfc69; no voters:
I20260925 02:47:15.183444 20457 raft_consensus.cc:2781] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:15.183597 20457 raft_consensus.cc:482] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:15.183699 20457 raft_consensus.cc:3037] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:15.185877 20457 raft_consensus.cc:504] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:15.186587 20457 leader_election.cc:288] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [CANDIDATE]: Term 1 election: Requested vote from peers 515334226e6e42c0a164b0f3ba5c646f (127.5.56.130:46035), f69cd492935f49f1bb9853c6c86bfc69 (127.5.56.131:45301)
I20260925 02:47:15.187045 20390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c0370656d1463ea7e832728c03a69b" candidate_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f69cd492935f49f1bb9853c6c86bfc69"
I20260925 02:47:15.187041 20258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01c0370656d1463ea7e832728c03a69b" candidate_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515334226e6e42c0a164b0f3ba5c646f"
I20260925 02:47:15.187261 20258 raft_consensus.cc:3037] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:15.187268 20390 raft_consensus.cc:3037] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:15.190582 20258 raft_consensus.cc:2454] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bcbc5eece8e4daf95cf58a96ec6dd71 in term 1.
I20260925 02:47:15.190582 20390 raft_consensus.cc:2454] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bcbc5eece8e4daf95cf58a96ec6dd71 in term 1.
I20260925 02:47:15.191030 20057 leader_election.cc:302] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515334226e6e42c0a164b0f3ba5c646f, 6bcbc5eece8e4daf95cf58a96ec6dd71; no voters:
I20260925 02:47:15.191360 20457 raft_consensus.cc:2781] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:15.191870 20457 raft_consensus.cc:686] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 1 LEADER]: Becoming Leader. State: Replica: 6bcbc5eece8e4daf95cf58a96ec6dd71, State: Running, Role: LEADER
I20260925 02:47:15.192458 20457 consensus_queue.cc:237] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } } peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } }
I20260925 02:47:15.197188 19982 catalog_manager.cc:5495] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bcbc5eece8e4daf95cf58a96ec6dd71 (127.5.56.129). New cstate: current_term: 1 leader_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 } health_report { overall_health: UNKNOWN } } }
I20260925 02:47:15.224615 20447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:15.219385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52038 (local address 127.5.56.129:41241)
0925 02:47:15.219587 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:15.219592 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:15.219610 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:47:15.219807 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:15.219810 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:15.219898 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:15.220023 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:15.220346 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.221071 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.222578 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.222984 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.223079 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:15.223529 (+ 450us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:15.224056 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:15.224168 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:15.224240 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 02:47:15.224308 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:47:15.235337 20390 raft_consensus.cc:1261] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69 [term 1 FOLLOWER]: Refusing update from remote peer 6bcbc5eece8e4daf95cf58a96ec6dd71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:15.235337 20258 raft_consensus.cc:1261] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f [term 1 FOLLOWER]: Refusing update from remote peer 6bcbc5eece8e4daf95cf58a96ec6dd71: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:15.236420 20462 consensus_queue.cc:1035] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "515334226e6e42c0a164b0f3ba5c646f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 46035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:15.236810 20457 consensus_queue.cc:1035] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f69cd492935f49f1bb9853c6c86bfc69" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 45301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:15.248416 20465 mvcc.cc:205] Tried to move back new op lower bound from 7333086966712823808 to 7333086966552461312. Current Snapshot: MvccSnapshot[applied={T|T < 7333086966712823808}]
I20260925 02:47:15.263603 19980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:15.263983 19980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38750:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:15.268899 19980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Sending DeleteTablet for 3 replicas of tablet 01c0370656d1463ea7e832728c03a69b
I20260925 02:47:15.270375 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 20019
I20260925 02:47:15.270532 20104 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c0370656d1463ea7e832728c03a69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:15 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:52036
I20260925 02:47:15.270746 20369 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c0370656d1463ea7e832728c03a69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:15 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:48654
I20260925 02:47:15.271646 20475 tablet_replica.cc:330] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: stopping tablet replica
I20260925 02:47:15.272537 20475 raft_consensus.cc:2229] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:47:15.272773 20237 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c0370656d1463ea7e832728c03a69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:15 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:33566
I20260925 02:47:15.273415 20475 raft_consensus.cc:2258] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:15.273371 20236 tablet_service.cc:1514] Processing DeleteTablet for tablet 01c0370656d1463ea7e832728c03a69b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:15 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:33566
I20260925 02:47:15.275694 20475 ts_tablet_manager.cc:1905] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:15.280239 20477 tablet_replica.cc:330] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: stopping tablet replica
I20260925 02:47:15.280673 20477 raft_consensus.cc:2229] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:15.281183 20477 raft_consensus.cc:2258] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:15.281972 20476 tablet_replica.cc:330] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: stopping tablet replica
I20260925 02:47:15.283083 20476 raft_consensus.cc:2229] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:15.283577 20476 raft_consensus.cc:2258] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:15.284293 20477 ts_tablet_manager.cc:1905] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:15.284334 20475 ts_tablet_manager.cc:1918] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:15.284544 20475 log.cc:1196] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/01c0370656d1463ea7e832728c03a69b
I20260925 02:47:15.285238 20475 ts_tablet_manager.cc:1939] T 01c0370656d1463ea7e832728c03a69b P 6bcbc5eece8e4daf95cf58a96ec6dd71: Deleting consensus metadata
I20260925 02:47:15.285723 20476 ts_tablet_manager.cc:1905] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:15.287218 19967 catalog_manager.cc:4841] TS 6bcbc5eece8e4daf95cf58a96ec6dd71 (127.5.56.129:41241): tablet 01c0370656d1463ea7e832728c03a69b (table test-table [id=66bcac8545274367969857b20c66de94]) successfully deleted
I20260925 02:47:15.291932 20477 ts_tablet_manager.cc:1918] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:15.292173 20477 log.cc:1196] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/01c0370656d1463ea7e832728c03a69b
I20260925 02:47:15.292831 20477 ts_tablet_manager.cc:1939] T 01c0370656d1463ea7e832728c03a69b P 515334226e6e42c0a164b0f3ba5c646f: Deleting consensus metadata
I20260925 02:47:15.295218 19968 catalog_manager.cc:4841] TS 515334226e6e42c0a164b0f3ba5c646f (127.5.56.130:46035): tablet 01c0370656d1463ea7e832728c03a69b (table test-table [id=66bcac8545274367969857b20c66de94]) successfully deleted
W20260925 02:47:15.295763 19968 catalog_manager.cc:4808] TS 515334226e6e42c0a164b0f3ba5c646f (127.5.56.130:46035): delete failed for tablet 01c0370656d1463ea7e832728c03a69b because the tablet was not found. No further retry: Not found: Tablet not found: 01c0370656d1463ea7e832728c03a69b
I20260925 02:47:15.296379 20476 ts_tablet_manager.cc:1918] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:15.296563 20476 log.cc:1196] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/01c0370656d1463ea7e832728c03a69b
I20260925 02:47:15.297175 20476 ts_tablet_manager.cc:1939] T 01c0370656d1463ea7e832728c03a69b P f69cd492935f49f1bb9853c6c86bfc69: Deleting consensus metadata
I20260925 02:47:15.298277 20447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:15.271573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52052 (local address 127.5.56.129:41241)
0925 02:47:15.271755 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:15.271763 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:15.271785 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:47:15.271850 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:15.271857 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:15.271938 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:15.272040 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:15.273772 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.274955 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.276825 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.277592 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.277758 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:15.289696 (+ 11938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:15.289709 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:15.289725 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:15.289772 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:15.293917 (+ 4145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:15.295237 (+ 1320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:15.295247 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:15.295251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:15.295416 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:15.295905 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:15.295910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:15.295913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:15.296448 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:15.296771 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:15.297566 (+ 795us) server_negotiation.cc:299] Negotiation successful
0925 02:47:15.297865 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:47:15.299100 19969 catalog_manager.cc:4841] TS f69cd492935f49f1bb9853c6c86bfc69 (127.5.56.131:45301): tablet 01c0370656d1463ea7e832728c03a69b (table test-table [id=66bcac8545274367969857b20c66de94]) successfully deleted
I20260925 02:47:15.299187 20165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:15.403503 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 20019
I20260925 02:47:15.423787 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 20174
I20260925 02:47:15.438194 20448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:15.424532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33574 (local address 127.5.56.130:46035)
0925 02:47:15.424740 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:15.424748 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:15.424822 (+ 74us) server_negotiation.cc:407] Connection header received
0925 02:47:15.425034 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:15.425038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:15.425120 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:15.425232 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:15.426676 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.427427 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.428814 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.429225 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.429377 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:15.432367 (+ 2990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:15.432384 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:15.432401 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:15.432458 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:15.435264 (+ 2806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:15.436017 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:15.436024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:15.436026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:15.436151 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:15.436538 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:15.436544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:15.436547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:15.436973 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:15.437295 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:15.437599 (+ 304us) server_negotiation.cc:299] Negotiation successful
0925 02:47:15.437831 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260925 02:47:15.439066 20298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:15.529692 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 20174
I20260925 02:47:15.547276 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 20306
I20260925 02:47:15.561107 20450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:15.548111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.5.56.131:45301)
0925 02:47:15.548299 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:15.548307 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:15.548396 (+ 89us) server_negotiation.cc:407] Connection header received
0925 02:47:15.548600 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:15.548606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:15.548680 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:15.548790 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:15.550216 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.551031 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.552317 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:15.552702 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:15.552833 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:15.555458 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:15.555472 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:15.555488 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:15.555521 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:15.558175 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:15.558955 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:15.558964 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:15.558967 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:15.559117 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:15.559515 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:15.559521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:15.559525 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:15.559905 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:15.560202 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:15.560530 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 02:47:15.560780 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:47:15.561818 20430 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:15.653549 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 20306
I20260925 02:47:15.671145 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 19949
I20260925 02:47:15.672330 20012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:15.790958 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 19949
I20260925 02:47:15.814129 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41657
--webserver_interface=127.5.56.190
--webserver_port=37409
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:41657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:16.157807 20482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:16.158146 20482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:16.158327 20482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:16.165753 20482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:16.165879 20482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:16.165978 20482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:16.166062 20482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:16.166131 20482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:16.176374 20482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:41657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=37409
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:16.178103 20482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:16.179669 20482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:16.192474 20488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:16.192425 20490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:16.192526 20487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:16.198570 20482 server_base.cc:1034] running on GCE node
I20260925 02:47:16.199407 20482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:16.200909 20482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:16.202121 20482 hybrid_clock.cc:648] HybridClock initialized: now 1790304436202075 us; error 38 us; skew 500 ppm
I20260925 02:47:16.202562 20482 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:16 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304436, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:16.205719 20482 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:47:16.207479 20482 webserver.cc:460] Webserver started at http://127.5.56.190:37409/ using document root <none> and password file <none>
I20260925 02:47:16.208243 20482 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:16.208396 20482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:16.214259 20482 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260925 02:47:16.217159 20497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:16.218614 20482 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:16.218802 20482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "f67d111f1c2845a4a117f2cb83d5eb64"
format_stamp: "Formatted at 2026-09-25 02:47:10 on dist-test-slave-n563"
server_key: "5df4bf6bb4374b102932869e59529785"
server_key_iv: "8d508034d72501bb7097ef75d34ee2c5"
server_key_version: "encryptionkey@0"
I20260925 02:47:16.219822 20482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:16.240240 20482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:16.244467 20482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:16.244753 20482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:16.264948 20482 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:41657
I20260925 02:47:16.264971 20549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:41657 every 8 connection(s)
I20260925 02:47:16.266590 20482 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:47:16.268563 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 20482
I20260925 02:47:16.269605 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:41241
--local_ip_for_outbound_sockets=127.5.56.129
--tserver_master_addrs=127.5.56.190:41657
--webserver_port=45909
--webserver_interface=127.5.56.129
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:47:16.276682 20550 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:47:16.281019 20550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Bootstrap starting.
I20260925 02:47:16.301985 20550 log.cc:824] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:16.318763 20550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:47:16.319367 20550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64: Bootstrap complete.
I20260925 02:47:16.329668 20550 raft_consensus.cc:348] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:16.330832 20550 raft_consensus.cc:729] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f67d111f1c2845a4a117f2cb83d5eb64, State: Initialized, Role: FOLLOWER
I20260925 02:47:16.331521 20550 consensus_queue.cc:260] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:16.331787 20550 raft_consensus.cc:388] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:47:16.331943 20550 raft_consensus.cc:482] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:47:16.332150 20550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:47:16.336429 20550 raft_consensus.cc:504] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:16.337042 20550 leader_election.cc:302] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f67d111f1c2845a4a117f2cb83d5eb64; no voters:
I20260925 02:47:16.337860 20550 leader_election.cc:288] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:47:16.338018 20554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:47:16.339156 20554 raft_consensus.cc:686] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [term 3 LEADER]: Becoming Leader. State: Replica: f67d111f1c2845a4a117f2cb83d5eb64, State: Running, Role: LEADER
I20260925 02:47:16.339812 20554 consensus_queue.cc:237] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } }
I20260925 02:47:16.340317 20550 sys_catalog.cc:564] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:16.342728 20555 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } } }
I20260925 02:47:16.343232 20555 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:16.347193 20556 sys_catalog.cc:455] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f67d111f1c2845a4a117f2cb83d5eb64. Latest consensus state: current_term: 3 leader_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f67d111f1c2845a4a117f2cb83d5eb64" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41657 } } }
I20260925 02:47:16.347585 20556 sys_catalog.cc:458] T 00000000000000000000000000000000 P f67d111f1c2845a4a117f2cb83d5eb64 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:16.349520 20560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:47:16.361310 20560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:47:16.363667 20560 catalog_manager.cc:1259] Loaded cluster ID: d06b4eec85da4c08899127f75c6d45d7
I20260925 02:47:16.363835 20560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 02:47:16.367094 20560 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f01456359ad google::LogMessage::Fail() at ??:0
@ 0x7f014563888a google::LogMessage::SendToLog() at ??:0
@ 0x7f01456354de google::LogMessage::Flush() at ??:0
@ 0x7f0145636a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f015b068512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f015b065976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f015b0ed610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f015b0ed439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0148281345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f014828491c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0148284799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f014825feaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f014b5396db start_thread at ??:0
@ 0x7f01444cf61f clone at ??:0
W20260925 02:47:16.700378 20552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:16.700695 20552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:16.700801 20552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:16.708547 20552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:16.708683 20552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:16.708811 20552 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:47:16.719619 20552 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:41241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=45909
--tserver_master_addrs=127.5.56.190:41657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:16.720996 20552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:16.722518 20552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:16.736384 20591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:16.736424 20594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:16.736346 20592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:16.737484 20552 server_base.cc:1034] running on GCE node
I20260925 02:47:16.738127 20552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:16.739233 20552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:16.740473 20552 hybrid_clock.cc:648] HybridClock initialized: now 1790304436740420 us; error 47 us; skew 500 ppm
I20260925 02:47:16.740821 20552 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:16 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304436, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:16.744055 20552 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:47:16.745637 20552 webserver.cc:460] Webserver started at http://127.5.56.129:45909/ using document root <none> and password file <none>
I20260925 02:47:16.746373 20552 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:16.746595 20552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:16.752092 20552 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:47:16.754954 20601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:16.756318 20552 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:16.756510 20552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "6bcbc5eece8e4daf95cf58a96ec6dd71"
format_stamp: "Formatted at 2026-09-25 02:47:10 on dist-test-slave-n563"
server_key: "3452aae20e2dc4e6c4deb931809e11ea"
server_key_iv: "1c4e4cc7aefb6c796bba20629e12aa9f"
server_key_version: "encryptionkey@0"
I20260925 02:47:16.757575 20552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:16.805394 20552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:16.809587 20552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:16.809903 20552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:16.811034 20552 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:16.812659 20552 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:16.812783 20552 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:16.812899 20552 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:16.813010 20552 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:16.848037 20552 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:41241
I20260925 02:47:16.848126 20714 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:41241 every 8 connection(s)
I20260925 02:47:16.849568 20552 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:47:16.850356 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 20552
I20260925 02:47:16.851282 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:46035
--local_ip_for_outbound_sockets=127.5.56.130
--tserver_master_addrs=127.5.56.190:41657
--webserver_port=46853
--webserver_interface=127.5.56.130
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:47:16.855273 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:16.852980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:51101)
0925 02:47:16.853853 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:16.854147 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":363,"threads_started":1}
W20260925 02:47:16.855997 20715 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:41657 (0 consecutive failures): Network error: Failed to ping master at 127.5.56.190:41657: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
I20260925 02:47:16.856906 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:16.856583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:45789)
0925 02:47:16.856728 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:16.856804 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 02:47:16.857681 20717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:16.857361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:51209)
0925 02:47:16.857499 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:16.857580 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:47:16.857889 20715 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:47:17.204648 20718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:17.205006 20718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:17.205197 20718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:17.212932 20718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:17.213095 20718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:17.213301 20718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:47:17.223992 20718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:46035
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=46853
--tserver_master_addrs=127.5.56.190:41657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:17.225481 20718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:17.227021 20718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:17.244287 20723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:17.244194 20724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:17.244270 20726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:17.244841 20718 server_base.cc:1034] running on GCE node
I20260925 02:47:17.245658 20718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:17.246731 20718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:17.248010 20718 hybrid_clock.cc:648] HybridClock initialized: now 1790304437247958 us; error 43 us; skew 500 ppm
I20260925 02:47:17.248389 20718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:17 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304437, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:17.251854 20718 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:47:17.253405 20718 webserver.cc:460] Webserver started at http://127.5.56.130:46853/ using document root <none> and password file <none>
I20260925 02:47:17.254158 20718 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:17.254326 20718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:17.259928 20718 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:47:17.263089 20733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:17.264510 20718 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:17.264736 20718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "515334226e6e42c0a164b0f3ba5c646f"
format_stamp: "Formatted at 2026-09-25 02:47:11 on dist-test-slave-n563"
server_key: "675b5b6699e730a6756319ee963ee9cd"
server_key_iv: "fb7911ae9e4d6f444d75e9be5bd00557"
server_key_version: "encryptionkey@0"
I20260925 02:47:17.265777 20718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:17.287921 20718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:17.292335 20718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:17.292644 20718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:17.293696 20718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:17.295120 20718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:17.295250 20718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:17.295392 20718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:17.295481 20718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:17.329806 20718 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:46035
I20260925 02:47:17.329897 20845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:46035 every 8 connection(s)
I20260925 02:47:17.331414 20718 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:47:17.336431 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:17.334530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:55343)
0925 02:47:17.335277 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:17.335515 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":285,"threads_started":1}
W20260925 02:47:17.337273 20846 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:41657 (0 consecutive failures): Network error: Failed to ping master at 127.5.56.190:41657: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
I20260925 02:47:17.337464 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 20718
I20260925 02:47:17.338090 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:45301
--local_ip_for_outbound_sockets=127.5.56.131
--tserver_master_addrs=127.5.56.190:41657
--webserver_port=46821
--webserver_interface=127.5.56.131
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:47:17.338140 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:17.337794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:59209)
0925 02:47:17.337955 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:17.338029 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 02:47:17.338985 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:17.338666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:34321)
0925 02:47:17.338814 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:17.338888 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 02:47:17.339226 20846 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:47:17.670728 20849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:17.671015 20849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:17.671110 20849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:17.678315 20849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:17.678515 20849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:17.678640 20849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:47:17.688606 20849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:45301
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=46821
--tserver_master_addrs=127.5.56.190:41657
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:17.689927 20849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:17.691256 20849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:17.704772 20854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:17.704828 20855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:17.704991 20857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:17.705809 20849 server_base.cc:1034] running on GCE node
I20260925 02:47:17.706524 20849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:17.707507 20849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:17.708735 20849 hybrid_clock.cc:648] HybridClock initialized: now 1790304437708660 us; error 69 us; skew 500 ppm
I20260925 02:47:17.709056 20849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:17 dist-test-slave-n563 krb5kdc[19388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304437, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:17.712462 20849 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:47:17.714020 20849 webserver.cc:460] Webserver started at http://127.5.56.131:46821/ using document root <none> and password file <none>
I20260925 02:47:17.714838 20849 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:17.714948 20849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:17.720382 20849 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260925 02:47:17.723172 20864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:17.724553 20849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:17.724740 20849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "f69cd492935f49f1bb9853c6c86bfc69"
format_stamp: "Formatted at 2026-09-25 02:47:11 on dist-test-slave-n563"
server_key: "fadde99d58df8627faeb9f94a0d6401a"
server_key_iv: "648518cdcd20b061a0ba03d169334cc5"
server_key_version: "encryptionkey@0"
I20260925 02:47:17.725768 20849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:17.769690 20849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:17.774191 20849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:17.774466 20849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:17.775480 20849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:17.776888 20849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:17.776976 20849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:17.777060 20849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:17.777114 20849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:17.811214 20849 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:45301
I20260925 02:47:17.811275 20976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:45301 every 8 connection(s)
I20260925 02:47:17.812852 20849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:47:17.814337 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 20849
I20260925 02:47:17.817898 20980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:17.816050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:41943)
0925 02:47:17.816782 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:17.817035 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":278,"threads_started":1}
W20260925 02:47:17.818657 20977 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:41657 (0 consecutive failures): Network error: Failed to ping master at 127.5.56.190:41657: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
I20260925 02:47:17.819476 20980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:17.819180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:39979)
0925 02:47:17.819312 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:17.819376 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 02:47:17.820230 20980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:17.819932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:40373)
0925 02:47:17.820052 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:17.820123 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:47:17.820472 20977 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:47:17.859587 20981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:17.858804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:44365)
0925 02:47:17.859265 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:17.859357 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":214,"threads_started":1}
I20260925 02:47:18.340588 20982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:18.339747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:60283)
0925 02:47:18.340257 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:18.340352 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":258,"threads_started":1}
I20260925 02:47:18.821929 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:18.821084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:49959)
0925 02:47:18.821562 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:18.821675 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":240,"threads_started":1}
I20260925 02:47:18.861336 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:18.860550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:39537)
0925 02:47:18.861027 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:18.861162 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":264,"threads_started":1}
I20260925 02:47:19.342326 20985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:19.341580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:33103)
0925 02:47:19.342062 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:19.342150 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":241,"threads_started":1}
I20260925 02:47:19.823681 20986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:19.822970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:35743)
0925 02:47:19.823422 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:19.823510 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":227,"threads_started":1}
I20260925 02:47:19.863148 20987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:19.862344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:58655)
0925 02:47:19.862882 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:19.862970 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":266,"threads_started":1}
I20260925 02:47:20.343930 20988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:20.343196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:59019)
0925 02:47:20.343661 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:20.343744 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":249,"threads_started":1}
I20260925 02:47:20.825522 20989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:20.824670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:56505)
0925 02:47:20.825249 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:20.825344 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":324,"threads_started":1}
I20260925 02:47:20.864769 20990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:20.864050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:36673)
0925 02:47:20.864501 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:20.864583 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":238,"threads_started":1}
I20260925 02:47:21.345832 20991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:21.345042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:60257)
0925 02:47:21.345526 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:21.345619 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":238,"threads_started":1}
I20260925 02:47:21.827164 20992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:21.826481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:44499)
0925 02:47:21.826907 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:21.826988 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":211,"threads_started":1}
I20260925 02:47:21.866735 20993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:21.865834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:35617)
0925 02:47:21.866331 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:21.866500 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":266,"threads_started":1}
I20260925 02:47:22.347613 20994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:22.346877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:47045)
0925 02:47:22.347361 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:22.347443 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":245,"threads_started":1}
I20260925 02:47:22.828914 20995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:22.828161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:51237)
0925 02:47:22.828649 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:22.828739 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":264,"threads_started":1}
I20260925 02:47:22.868425 20996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:22.867674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:37293)
0925 02:47:22.868149 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:22.868254 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":265,"threads_started":1}
I20260925 02:47:23.349382 20997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:23.348608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:37401)
0925 02:47:23.349092 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:23.349188 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":254,"threads_started":1}
I20260925 02:47:23.830741 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:23.829947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:58189)
0925 02:47:23.830462 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:23.830569 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":243,"threads_started":1}
I20260925 02:47:23.870370 20999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:23.869453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:46659)
0925 02:47:23.870018 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:23.870118 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":243,"threads_started":1}
I20260925 02:47:24.351212 21000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:24.350343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:59683)
0925 02:47:24.350906 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:24.350993 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":209,"threads_started":1}
I20260925 02:47:24.832458 21001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:24.831642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:38021)
0925 02:47:24.832146 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:24.832226 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":200,"threads_started":1}
I20260925 02:47:24.872139 21002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:24.871401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:36151)
0925 02:47:24.871883 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:24.871972 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":245,"threads_started":1}
I20260925 02:47:25.353199 21003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:25.352393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:47417)
0925 02:47:25.352926 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:25.353023 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":286,"threads_started":1}
I20260925 02:47:25.834053 21004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:25.833329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:58275)
0925 02:47:25.833807 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:25.833887 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":244,"threads_started":1}
I20260925 02:47:25.873909 21005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:25.873130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:58127)
0925 02:47:25.873626 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:25.873714 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":263,"threads_started":1}
I20260925 02:47:26.355132 21006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:26.354245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:45695)
0925 02:47:26.354856 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:26.354939 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":339,"threads_started":1}
I20260925 02:47:26.835978 21007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:26.835198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:58815)
0925 02:47:26.835690 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:26.835795 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":235,"threads_started":1}
I20260925 02:47:26.875870 21008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:26.874982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:58085)
0925 02:47:26.875522 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:26.875631 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":288,"threads_started":1}
I20260925 02:47:27.356976 21009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:27.356148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:33829)
0925 02:47:27.356684 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:27.356784 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":290,"threads_started":1}
I20260925 02:47:27.837898 21010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:27.837128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:44803)
0925 02:47:27.837626 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:27.837724 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":259,"threads_started":1}
I20260925 02:47:27.877697 21011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:27.876890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:42821)
0925 02:47:27.877397 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:27.877495 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":254,"threads_started":1}
I20260925 02:47:28.358832 21012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:28.358014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:32957)
0925 02:47:28.358536 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:28.358645 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":216,"threads_started":1}
I20260925 02:47:28.839835 21013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:28.838983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:49969)
0925 02:47:28.839544 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:28.839636 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":329,"threads_started":1}
I20260925 02:47:28.879487 21014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:28.878782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:38065)
0925 02:47:28.879222 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:28.879322 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":221,"threads_started":1}
I20260925 02:47:29.360682 21015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:29.359845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:46475)
0925 02:47:29.360389 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:29.360473 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":301,"threads_started":1}
I20260925 02:47:29.841733 21016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:29.840946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:45357)
0925 02:47:29.841421 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:29.841509 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":251,"threads_started":1}
I20260925 02:47:29.881198 21017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:29.880429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:46573)
0925 02:47:29.880904 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:29.881002 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":244,"threads_started":1}
I20260925 02:47:30.362690 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:30.361727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:59867)
0925 02:47:30.362336 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:30.362496 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":374,"threads_started":1}
I20260925 02:47:30.843585 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:30.842745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:41239)
0925 02:47:30.843272 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:30.843378 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":244,"threads_started":1}
I20260925 02:47:30.883109 21020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:30.882248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:34399)
0925 02:47:30.882843 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:30.882923 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":341,"threads_started":1}
I20260925 02:47:31.364573 21021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:31.363728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:53379)
0925 02:47:31.364288 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:31.364382 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":309,"threads_started":1}
I20260925 02:47:31.845355 21022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:31.844612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:35899)
0925 02:47:31.845060 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:31.845141 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":225,"threads_started":1}
I20260925 02:47:31.885082 21023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:31.884334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:54711)
0925 02:47:31.884784 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:31.884905 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":224,"threads_started":1}
I20260925 02:47:32.366679 21024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:32.365883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.130:42959)
0925 02:47:32.366312 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:32.366487 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":215,"threads_started":1}
I20260925 02:47:32.819321 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 20552
I20260925 02:47:32.847470 21026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:32.846658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.131:51787)
0925 02:47:32.847189 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:32.847287 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":293,"threads_started":1}
I20260925 02:47:32.854640 21025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:32.820620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33888 (local address 127.5.56.129:41241)
0925 02:47:32.821459 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:32.821480 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:32.821533 (+ 53us) server_negotiation.cc:407] Connection header received
0925 02:47:32.823655 (+ 2122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:32.823665 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:32.824114 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:32.824430 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:32.825917 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:32.827331 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:32.828657 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:32.829378 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:32.829583 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:32.846649 (+ 17066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:32.846663 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:32.846684 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:32.846741 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:32.850584 (+ 3843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:32.851462 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:32.851471 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:32.851479 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:32.851729 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:32.852231 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:32.852243 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:32.852247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:32.852882 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:32.853243 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:32.853778 (+ 535us) server_negotiation.cc:299] Negotiation successful
0925 02:47:32.854256 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":540,"threads_started":1}
I20260925 02:47:32.856142 20710 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:32.944280 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 20552
I20260925 02:47:32.944348 21028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:32.943174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:41657 (local address 127.5.56.129:56473)
0925 02:47:32.943713 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:32.943956 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:41657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":244,"threads_started":1}
I20260925 02:47:32.963821 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 20718
I20260925 02:47:32.982295 21029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:32.964943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46142 (local address 127.5.56.130:46035)
0925 02:47:32.965478 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:32.965492 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:32.965530 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:47:32.967209 (+ 1679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:32.967219 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:32.967662 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:32.967955 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:32.969326 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:32.970820 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:32.972079 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:32.972818 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:32.972951 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:32.975847 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:32.975856 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:32.975873 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:32.975916 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:32.978938 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:32.979660 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:32.979668 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:32.979675 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:32.979854 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:32.980232 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:32.980236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:32.980239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:32.980886 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:32.981149 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:32.981536 (+ 387us) server_negotiation.cc:299] Negotiation successful
0925 02:47:32.981940 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":271,"threads_started":1}
I20260925 02:47:32.983850 20841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:33.070545 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 20718
I20260925 02:47:33.088550 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 20849
I20260925 02:47:33.110050 21031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:33.089571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43258 (local address 127.5.56.131:45301)
0925 02:47:33.090114 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:33.090128 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:33.090171 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:47:33.092650 (+ 2479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:33.092666 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:33.093275 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:33.095658 (+ 2383us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:33.096256 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:33.098103 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:33.099817 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:33.100544 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:33.100711 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:33.103647 (+ 2936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:33.103655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:33.103674 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:33.103748 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:33.106649 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:33.107395 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:33.107404 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:33.107411 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:33.107661 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:33.108046 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:33.108051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:33.108054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:33.108692 (+ 638us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:33.108953 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:33.109373 (+ 420us) server_negotiation.cc:299] Negotiation successful
0925 02:47:33.109736 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":242,"threads_started":1}
I20260925 02:47:33.111559 20972 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:33.197546 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 20849
2026-09-25T02:47:33Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24836 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:47:33 dist-test-slave-n563 krb5kdc[21035](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:47:33 dist-test-slave-n563 krb5kdc[21035](info): set up 2 sockets
Sep 25 02:47:33 dist-test-slave-n563 krb5kdc[21035](info): commencing operation
krb5kdc: starting...
W20260925 02:47:34.324579 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.007s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:47:34 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:47:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:47:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:47:34.589778 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:35191
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:35191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:34.930632 21048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:34.931022 21048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:34.931195 21048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:34.938513 21048 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:47:34.938675 21048 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:47:34.938746 21048 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:47:34.938831 21048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:34.938937 21048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:34.939064 21048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:34.939155 21048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:34.939237 21048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:34.949455 21048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:35191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:35191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:34.950914 21048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:34.952309 21048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:34.964883 21053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:34.965061 21054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:34.965065 21056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:34.965652 21048 server_base.cc:1034] running on GCE node
I20260925 02:47:34.966607 21048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:34.968155 21048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:34.969516 21048 hybrid_clock.cc:648] HybridClock initialized: now 1790304454969399 us; error 84 us; skew 500 ppm
I20260925 02:47:34.969920 21048 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:34 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304454, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:34.973161 21048 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:47:34.974691 21048 webserver.cc:460] Webserver started at http://127.5.56.190:38359/ using document root <none> and password file <none>
I20260925 02:47:34.975350 21048 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:34.975431 21048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:34.975693 21048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:34.978776 21048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "61af38f9527c4033a0754330e44a9572"
format_stamp: "Formatted at 2026-09-25 02:47:34 on dist-test-slave-n563"
server_key: "2163a5ef27497465f29f7caeffb542e1"
server_key_iv: "768051f70cf397f440c1ac394391754f"
server_key_version: "encryptionkey@0"
I20260925 02:47:34.979535 21048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "61af38f9527c4033a0754330e44a9572"
format_stamp: "Formatted at 2026-09-25 02:47:34 on dist-test-slave-n563"
server_key: "2163a5ef27497465f29f7caeffb542e1"
server_key_iv: "768051f70cf397f440c1ac394391754f"
server_key_version: "encryptionkey@0"
I20260925 02:47:34.984756 21048 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:47:34.988207 21063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:34.989796 21048 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:47:34.990012 21048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "61af38f9527c4033a0754330e44a9572"
format_stamp: "Formatted at 2026-09-25 02:47:34 on dist-test-slave-n563"
server_key: "2163a5ef27497465f29f7caeffb542e1"
server_key_iv: "768051f70cf397f440c1ac394391754f"
server_key_version: "encryptionkey@0"
I20260925 02:47:34.990240 21048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:35.030186 21048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:35.034821 21048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:35.035141 21048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:35.055670 21048 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:35191
I20260925 02:47:35.055711 21114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:35191 every 8 connection(s)
I20260925 02:47:35.057125 21048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:47:35.061048 21115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:35.065780 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 21048
I20260925 02:47:35.066075 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:47:35.066865 5346 external_mini_cluster.cc:1428] Setting key 0b498fc50d635e4fd8b55684d59f68cb
I20260925 02:47:35.072379 21115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Bootstrap starting.
I20260925 02:47:35.076243 21115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:35.077528 21115 log.cc:824] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:35.081238 21115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: No bootstrap required, opened a new log
Sep 25 02:47:35 dist-test-slave-n563 krb5kdc[21035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:35.092190 21115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:47:35.092689 21115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:35.092825 21115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61af38f9527c4033a0754330e44a9572, State: Initialized, Role: FOLLOWER
I20260925 02:47:35.093564 21115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:47:35.093806 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:35.068995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33904 (local address 127.5.56.190:35191)
0925 02:47:35.069933 (+ 938us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:35.069946 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:35.069997 (+ 51us) server_negotiation.cc:407] Connection header received
0925 02:47:35.071838 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:35.071864 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:35.072296 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:35.072732 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:35.074599 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:35.075941 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:35.077923 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:35.079144 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:35.079281 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:35.084358 (+ 5077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:35.084369 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:35.084396 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:35.084450 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:35.088760 (+ 4310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:35.089498 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:35.089509 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:35.089524 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:35.089786 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:35.090166 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:35.090179 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:35.090183 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:35.091033 (+ 850us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:35.091323 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:35.091735 (+ 412us) server_negotiation.cc:299] Negotiation successful
0925 02:47:35.092232 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":439,"threads_started":1}
I20260925 02:47:35.093879 21115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:47:35.094060 21115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:47:35.094285 21115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:35.096778 21115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:47:35.097322 21115 leader_election.cc:302] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61af38f9527c4033a0754330e44a9572; no voters:
I20260925 02:47:35.097920 21115 leader_election.cc:288] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:47:35.098074 21120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:35.098901 21120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 1 LEADER]: Becoming Leader. State: Replica: 61af38f9527c4033a0754330e44a9572, State: Running, Role: LEADER
I20260925 02:47:35.099502 21120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:47:35.100167 21115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:35.102159 21122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61af38f9527c4033a0754330e44a9572. Latest consensus state: current_term: 1 leader_uuid: "61af38f9527c4033a0754330e44a9572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } } }
I20260925 02:47:35.102617 21122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:35.103358 21121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "61af38f9527c4033a0754330e44a9572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } } }
I20260925 02:47:35.103681 21121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:35.103869 21125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:47:35.112634 21125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:47:35.122308 21125 catalog_manager.cc:1347] Generated new cluster ID: 9d6d6b5d738c423ba1cf6cb9a1064d79
I20260925 02:47:35.122575 21125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:47:35.130699 21125 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:47:35.132061 21125 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:47:35.143450 21125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Generated new TSK 0
I20260925 02:47:35.144395 21125 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:47:35.226702 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:35191
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:35.580360 21141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:35.580654 21141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:35.580790 21141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:35.588443 21141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:35.588593 21141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:35.588773 21141 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:47:35.599731 21141 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:35.601213 21141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:35.602665 21141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:35.616283 21149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:35.616317 21147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:35.616230 21146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:35.617494 21141 server_base.cc:1034] running on GCE node
I20260925 02:47:35.618185 21141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:35.619292 21141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:35.620539 21141 hybrid_clock.cc:648] HybridClock initialized: now 1790304455620490 us; error 43 us; skew 500 ppm
I20260925 02:47:35.620895 21141 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:35 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304455, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:35.624197 21141 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:47:35.625795 21141 webserver.cc:460] Webserver started at http://127.5.56.129:34599/ using document root <none> and password file <none>
I20260925 02:47:35.626592 21141 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:35.626746 21141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:35.627081 21141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:35.630154 21141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "5e899bd70a7f4fa3908a10a38a65e274"
format_stamp: "Formatted at 2026-09-25 02:47:35 on dist-test-slave-n563"
server_key: "d904ddf3ed3a1037a4e49ab70156ee7a"
server_key_iv: "7c0456b6efada8451ff44706235889f6"
server_key_version: "encryptionkey@0"
I20260925 02:47:35.631016 21141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "5e899bd70a7f4fa3908a10a38a65e274"
format_stamp: "Formatted at 2026-09-25 02:47:35 on dist-test-slave-n563"
server_key: "d904ddf3ed3a1037a4e49ab70156ee7a"
server_key_iv: "7c0456b6efada8451ff44706235889f6"
server_key_version: "encryptionkey@0"
I20260925 02:47:35.637225 21141 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:47:35.640947 21156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:35.642506 21141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:47:35.642649 21141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "5e899bd70a7f4fa3908a10a38a65e274"
format_stamp: "Formatted at 2026-09-25 02:47:35 on dist-test-slave-n563"
server_key: "d904ddf3ed3a1037a4e49ab70156ee7a"
server_key_iv: "7c0456b6efada8451ff44706235889f6"
server_key_version: "encryptionkey@0"
I20260925 02:47:35.642832 21141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:35.655256 21141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:35.659457 21141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:35.659721 21141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:35.660647 21141 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:35.662097 21141 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:35.662173 21141 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:35.662259 21141 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:35.662303 21141 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:35.697139 21141 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:36505
I20260925 02:47:35.697233 21269 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:36505 every 8 connection(s)
I20260925 02:47:35.698520 21141 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:47:35.706247 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 21141
I20260925 02:47:35.706518 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:47:35.707204 5346 external_mini_cluster.cc:1428] Setting key f32ef7d9c7103a1d8eceb09d2b7cc450
Sep 25 02:47:35 dist-test-slave-n563 krb5kdc[21035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304455, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:35.721208 21272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:35.701168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:41553 (local address 127.5.56.190:35191)
0925 02:47:35.701691 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:35.701696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:35.702832 (+ 1136us) server_negotiation.cc:407] Connection header received
0925 02:47:35.704748 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:35.704754 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:35.704843 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:35.704973 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:35.706595 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:35.707384 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:35.709217 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:35.709639 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:35.709767 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:35.713806 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:35.713816 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:35.713824 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:35.713870 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:35.717549 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:35.718690 (+ 1141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:35.718694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:35.718697 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:35.718824 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:35.719384 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:35.719387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:35.719389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:35.719671 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:35.719788 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:35.720756 (+ 968us) server_negotiation.cc:299] Negotiation successful
0925 02:47:35.720937 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":264,"threads_started":1}
I20260925 02:47:35.722826 21273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:35.701799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:41553)
0925 02:47:35.702667 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:35.702714 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:35.704460 (+ 1746us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:35.705222 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:35.705231 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:35.705784 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:35.706353 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:35.706371 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:35.707555 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:35.707567 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:35.709080 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:35.709099 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:35.709901 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:35.709909 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:35.710134 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:35.710926 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:35.710953 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:35.713547 (+ 2594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:35.717714 (+ 4167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:35.717721 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:35.717736 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:35.718516 (+ 780us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:35.718958 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:35.718966 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:35.718972 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:35.719271 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:35.719804 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:35.719815 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:35.720314 (+ 499us) client_negotiation.cc:769] Sending connection context
0925 02:47:35.720630 (+ 316us) client_negotiation.cc:240] Negotiation successful
0925 02:47:35.721257 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":305,"threads_started":1}
I20260925 02:47:35.724485 21270 heartbeater.cc:347] Connected to a master server at 127.5.56.190:35191
I20260925 02:47:35.724861 21270 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:35.725593 21270 heartbeater.cc:511] Master 127.5.56.190:35191 requested a full tablet report, sending...
I20260925 02:47:35.727795 21080 ts_manager.cc:194] Registered new tserver with Master: 5e899bd70a7f4fa3908a10a38a65e274 (127.5.56.129:36505)
I20260925 02:47:35.729315 21080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:41553
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:47:35.767172 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:35191
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:36.115033 21276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:36.115381 21276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:36.115550 21276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:36.123255 21276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:36.123380 21276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:36.123575 21276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:47:36.133908 21276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:36.135313 21276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:36.136672 21276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:36.150765 21284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:36.150700 21281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:36.150730 21282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:36.151566 21276 server_base.cc:1034] running on GCE node
I20260925 02:47:36.152258 21276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:36.153236 21276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:36.154490 21276 hybrid_clock.cc:648] HybridClock initialized: now 1790304456154444 us; error 57 us; skew 500 ppm
I20260925 02:47:36.154817 21276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:36 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304456, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:36.157990 21276 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:47:36.159574 21276 webserver.cc:460] Webserver started at http://127.5.56.130:39993/ using document root <none> and password file <none>
I20260925 02:47:36.160301 21276 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:36.160451 21276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:36.160780 21276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:36.164516 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c"
format_stamp: "Formatted at 2026-09-25 02:47:36 on dist-test-slave-n563"
server_key: "ef27e4405c32059065ab69023f4e92b5"
server_key_iv: "c519912d6ebf4c868706c25226349390"
server_key_version: "encryptionkey@0"
I20260925 02:47:36.165323 21276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c"
format_stamp: "Formatted at 2026-09-25 02:47:36 on dist-test-slave-n563"
server_key: "ef27e4405c32059065ab69023f4e92b5"
server_key_iv: "c519912d6ebf4c868706c25226349390"
server_key_version: "encryptionkey@0"
I20260925 02:47:36.170872 21276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:47:36.174721 21292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:36.176199 21276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:47:36.176400 21276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c"
format_stamp: "Formatted at 2026-09-25 02:47:36 on dist-test-slave-n563"
server_key: "ef27e4405c32059065ab69023f4e92b5"
server_key_iv: "c519912d6ebf4c868706c25226349390"
server_key_version: "encryptionkey@0"
I20260925 02:47:36.176616 21276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:36.206779 21276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:36.211040 21276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:36.211380 21276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:36.212387 21276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:36.213886 21276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:36.213960 21276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:36.214092 21276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:36.214150 21276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:36.248466 21276 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:39845
I20260925 02:47:36.248518 21405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:39845 every 8 connection(s)
I20260925 02:47:36.249861 21276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:47:36.254170 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 21276
I20260925 02:47:36.254377 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:47:36.254894 5346 external_mini_cluster.cc:1428] Setting key c50dce6a76182fba4f8143281564b89f
Sep 25 02:47:36 dist-test-slave-n563 krb5kdc[21035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304456, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:36.270644 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.252459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:36421 (local address 127.5.56.190:35191)
0925 02:47:36.252950 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:36.252957 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:36.253889 (+ 932us) server_negotiation.cc:407] Connection header received
0925 02:47:36.255925 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:36.255931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:36.256026 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:36.256148 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:36.257812 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.258610 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.260184 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.260632 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.260774 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.264618 (+ 3844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:36.264623 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:36.264627 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:36.264661 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:36.267396 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:36.268173 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:36.268179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:36.268182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:36.268325 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:36.268785 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:36.268796 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:36.268799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:36.269067 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:36.269242 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:36.269987 (+ 745us) server_negotiation.cc:299] Negotiation successful
0925 02:47:36.270236 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":358,"thread_start_us":234,"threads_started":1}
I20260925 02:47:36.271711 21409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.253014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:36421)
0925 02:47:36.253718 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:36.253761 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:36.255648 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:36.256410 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:36.256418 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:36.257046 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:36.257623 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.257641 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.258772 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.258776 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.260031 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.260043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.260870 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.260880 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.261121 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.261833 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:36.261856 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:36.264361 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:36.267564 (+ 3203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:36.267574 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:36.267593 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:36.268043 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:36.268443 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:36.268449 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:36.268452 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:36.268658 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:36.269204 (+ 546us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:36.269213 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:36.269625 (+ 412us) client_negotiation.cc:769] Sending connection context
0925 02:47:36.269904 (+ 279us) client_negotiation.cc:240] Negotiation successful
0925 02:47:36.270531 (+ 627us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":291,"threads_started":1}
I20260925 02:47:36.273310 21406 heartbeater.cc:347] Connected to a master server at 127.5.56.190:35191
I20260925 02:47:36.273659 21406 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:36.274610 21406 heartbeater.cc:511] Master 127.5.56.190:35191 requested a full tablet report, sending...
I20260925 02:47:36.276396 21080 ts_manager.cc:194] Registered new tserver with Master: 8cd4db00a9174eb8b9cf2fb6e1cd392c (127.5.56.130:39845)
I20260925 02:47:36.277627 21080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:36421
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:47:36.311728 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:35191
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:36.654495 21412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:36.654919 21412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:36.655025 21412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:36.662482 21412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:36.662606 21412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:36.662730 21412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:47:36.674206 21412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:36.676108 21412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:36.677870 21412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:36.693635 21420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:36.693684 21418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:36.693696 21417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:36.694515 21412 server_base.cc:1034] running on GCE node
I20260925 02:47:36.695236 21412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:36.696339 21412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:36.697598 21412 hybrid_clock.cc:648] HybridClock initialized: now 1790304456697524 us; error 62 us; skew 500 ppm
I20260925 02:47:36.697993 21412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:36 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304456, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:36.702108 21412 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:47:36.703908 21412 webserver.cc:460] Webserver started at http://127.5.56.131:33929/ using document root <none> and password file <none>
I20260925 02:47:36.704574 21412 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:36.704676 21412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:36.704949 21412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:47:36.708087 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "76fbd1d0e43f4bef871584cb9308a3d7"
format_stamp: "Formatted at 2026-09-25 02:47:36 on dist-test-slave-n563"
server_key: "2b3fc311f04964d886286322306f4f0b"
server_key_iv: "da1bdd9104fa8d462947fb2c0d90b1e0"
server_key_version: "encryptionkey@0"
I20260925 02:47:36.709209 21412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "76fbd1d0e43f4bef871584cb9308a3d7"
format_stamp: "Formatted at 2026-09-25 02:47:36 on dist-test-slave-n563"
server_key: "2b3fc311f04964d886286322306f4f0b"
server_key_iv: "da1bdd9104fa8d462947fb2c0d90b1e0"
server_key_version: "encryptionkey@0"
I20260925 02:47:36.714670 21412 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 02:47:36.718264 21427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:36.719681 21412 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:47:36.719882 21412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "76fbd1d0e43f4bef871584cb9308a3d7"
format_stamp: "Formatted at 2026-09-25 02:47:36 on dist-test-slave-n563"
server_key: "2b3fc311f04964d886286322306f4f0b"
server_key_iv: "da1bdd9104fa8d462947fb2c0d90b1e0"
server_key_version: "encryptionkey@0"
I20260925 02:47:36.720110 21412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:36.732481 21270 heartbeater.cc:503] Master 127.5.56.190:35191 was elected leader, sending a full tablet report...
I20260925 02:47:36.739029 21412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:36.743409 21412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:36.743727 21412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:36.744688 21412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:36.746016 21412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:36.746089 21412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:36.746220 21412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:36.746357 21412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:36.780664 21412 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:46183
I20260925 02:47:36.780766 21540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:46183 every 8 connection(s)
I20260925 02:47:36.782078 21412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:47:36.787371 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 21412
I20260925 02:47:36.787570 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:47:36.788134 5346 external_mini_cluster.cc:1428] Setting key 0115e93bda634ef2ac0249081a456521
Sep 25 02:47:36 dist-test-slave-n563 krb5kdc[21035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304456, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:36.804991 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.784875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:45751 (local address 127.5.56.190:35191)
0925 02:47:36.785395 (+ 520us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:36.785400 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:36.786241 (+ 841us) server_negotiation.cc:407] Connection header received
0925 02:47:36.788803 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:36.788809 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:36.788912 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:36.789036 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:36.791139 (+ 2103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.792132 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.793809 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.794190 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.794294 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.797973 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:36.797977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:36.797981 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:36.798015 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:36.801261 (+ 3246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:36.802187 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:36.802192 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:36.802194 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:36.802402 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:36.802957 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:36.802964 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:36.802969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:36.803354 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:36.803518 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:36.804520 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0925 02:47:36.804702 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":282,"threads_started":1}
I20260925 02:47:36.806139 21544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.785294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:45751)
0925 02:47:36.786055 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:36.786100 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:36.788438 (+ 2338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:36.789336 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:36.789344 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:36.790230 (+ 886us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:36.790969 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.790986 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.792281 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.792285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.793689 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.793701 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.794537 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.794543 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.794841 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.795486 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:36.795515 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:36.797761 (+ 2246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:36.801413 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:36.801421 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:36.801435 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:36.802049 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:36.802535 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:36.802540 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:36.802545 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:36.802833 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:36.803511 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:36.803518 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:36.804055 (+ 537us) client_negotiation.cc:769] Sending connection context
0925 02:47:36.804415 (+ 360us) client_negotiation.cc:240] Negotiation successful
0925 02:47:36.804884 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":301,"threads_started":1}
I20260925 02:47:36.807850 21541 heartbeater.cc:347] Connected to a master server at 127.5.56.190:35191
I20260925 02:47:36.808214 21541 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:36.809218 21541 heartbeater.cc:511] Master 127.5.56.190:35191 requested a full tablet report, sending...
I20260925 02:47:36.810952 21080 ts_manager.cc:194] Registered new tserver with Master: 76fbd1d0e43f4bef871584cb9308a3d7 (127.5.56.131:46183)
I20260925 02:47:36.811789 21080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:45751
I20260925 02:47:36.816669 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:47:36.833922 21543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.821177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33910 (local address 127.5.56.190:35191)
0925 02:47:36.821325 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:36.821329 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:36.821823 (+ 494us) server_negotiation.cc:407] Connection header received
0925 02:47:36.821966 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:36.821970 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:36.822037 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:36.822117 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:36.823715 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.824462 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.825908 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.826315 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.826484 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.828891 (+ 2407us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:36.828895 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:36.828899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:36.828927 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:36.831492 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:36.832145 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:36.832149 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:36.832151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:36.832284 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:36.832622 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:36.832625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:36.832627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:36.832945 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:36.833130 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:36.833411 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 02:47:36.833584 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:47:36.838223 21080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33910:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:47:36.845273 21080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:36.884056 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.858274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.5.56.130:39845)
0925 02:47:36.873641 (+ 15367us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:36.873651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:36.873685 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:47:36.873797 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:36.873809 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:36.874052 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:36.874255 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:36.876453 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.877998 (+ 1545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.880608 (+ 2610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.882195 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.882343 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.882603 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:36.883531 (+ 928us) server_negotiation.cc:299] Negotiation successful
0925 02:47:36.883704 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15120,"thread_start_us":16345,"threads_started":1}
I20260925 02:47:36.884114 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.873928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44364 (local address 127.5.56.129:36505)
0925 02:47:36.875959 (+ 2031us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:36.875969 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:36.876392 (+ 423us) server_negotiation.cc:407] Connection header received
0925 02:47:36.876638 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:36.876646 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:36.876878 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:36.877061 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:36.877849 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.879486 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.881604 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.883019 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.883190 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.883433 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:36.883646 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 02:47:36.883794 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1835,"thread_start_us":1637,"threads_started":1}
I20260925 02:47:36.884135 21559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.873549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36505 (local address 127.0.0.1:44364)
0925 02:47:36.876253 (+ 2704us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:36.876270 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:36.876397 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:36.877081 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:36.877086 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:36.877116 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:36.877596 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.877638 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.879657 (+ 2019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.879662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.881459 (+ 1797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.881472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.883255 (+ 1783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.883262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.883444 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.883481 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 02:47:36.883593 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 02:47:36.883719 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2560,"thread_start_us":2887,"threads_started":1}
I20260925 02:47:36.884626 21554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.858365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:46183 (local address 127.0.0.1:56864)
0925 02:47:36.872901 (+ 14536us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:36.872918 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:36.873060 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:36.874993 (+ 1933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:36.875004 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:36.875086 (+ 82us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:36.875615 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.875640 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.877327 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.877332 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.880249 (+ 2917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.880263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.882807 (+ 2544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.882813 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.883039 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.883196 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 02:47:36.883343 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 02:47:36.883590 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14363,"mutex_wait_us":165,"spinlock_wait_cycles":34688,"thread_start_us":13468,"threads_started":1}
I20260925 02:47:36.884845 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.858963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56864 (local address 127.5.56.131:46183)
0925 02:47:36.871368 (+ 12405us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:36.871382 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:36.873035 (+ 1653us) server_negotiation.cc:407] Connection header received
0925 02:47:36.873218 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:36.873225 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:36.873667 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:36.873888 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:36.875814 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.877196 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.880456 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.882195 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.882343 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.882650 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:36.884198 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0925 02:47:36.884417 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12111,"thread_start_us":12426,"threads_started":1}
I20260925 02:47:36.886340 21556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.858121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39845 (local address 127.0.0.1:56868)
0925 02:47:36.872423 (+ 14302us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:36.872491 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:36.872812 (+ 321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:36.874996 (+ 2184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:36.875005 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:36.875081 (+ 76us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:36.876246 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.876259 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.878169 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.878179 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.880421 (+ 2242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.880435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.882757 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.882766 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.883074 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.883196 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 02:47:36.883343 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 02:47:36.886028 (+ 2685us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13916,"mutex_wait_us":1984,"spinlock_wait_cycles":2304,"thread_start_us":15018,"threads_started":1}
I20260925 02:47:36.888366 21204 tablet_service.cc:1467] Processing CreateTablet for tablet c63ec70fcc4846f989a958175c2c3082 (DEFAULT_TABLE table=test-table [id=d778ddb6c6b94b3ca0926f3ce371e3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:36.888574 21476 tablet_service.cc:1467] Processing CreateTablet for tablet c63ec70fcc4846f989a958175c2c3082 (DEFAULT_TABLE table=test-table [id=d778ddb6c6b94b3ca0926f3ce371e3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:36.889444 21340 tablet_service.cc:1467] Processing CreateTablet for tablet c63ec70fcc4846f989a958175c2c3082 (DEFAULT_TABLE table=test-table [id=d778ddb6c6b94b3ca0926f3ce371e3a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:36.890021 21476 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63ec70fcc4846f989a958175c2c3082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:36.890022 21204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63ec70fcc4846f989a958175c2c3082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:36.890769 21340 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c63ec70fcc4846f989a958175c2c3082. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:36.904778 21560 tablet_bootstrap.cc:492] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Bootstrap starting.
I20260925 02:47:36.907243 21562 tablet_bootstrap.cc:492] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7: Bootstrap starting.
I20260925 02:47:36.909142 21560 tablet_bootstrap.cc:654] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:36.911139 21562 tablet_bootstrap.cc:654] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:36.911342 21560 log.cc:824] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:36.912261 21561 tablet_bootstrap.cc:492] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: Bootstrap starting.
I20260925 02:47:36.912603 21562 log.cc:824] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:36.915827 21560 tablet_bootstrap.cc:492] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: No bootstrap required, opened a new log
I20260925 02:47:36.915944 21561 tablet_bootstrap.cc:654] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:36.916177 21560 ts_tablet_manager.cc:1397] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20260925 02:47:36.916253 21562 tablet_bootstrap.cc:492] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7: No bootstrap required, opened a new log
I20260925 02:47:36.916512 21562 ts_tablet_manager.cc:1397] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20260925 02:47:36.917196 21561 log.cc:824] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:36.920600 21561 tablet_bootstrap.cc:492] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: No bootstrap required, opened a new log
I20260925 02:47:36.920976 21561 ts_tablet_manager.cc:1397] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 02:47:36.925010 21562 raft_consensus.cc:348] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:36.925428 21562 raft_consensus.cc:374] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:36.925499 21562 raft_consensus.cc:729] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fbd1d0e43f4bef871584cb9308a3d7, State: Initialized, Role: FOLLOWER
I20260925 02:47:36.926055 21562 consensus_queue.cc:260] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:36.927171 21541 heartbeater.cc:503] Master 127.5.56.190:35191 was elected leader, sending a full tablet report...
I20260925 02:47:36.927426 21560 raft_consensus.cc:348] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:36.927881 21562 ts_tablet_manager.cc:1428] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260925 02:47:36.927915 21560 raft_consensus.cc:374] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:36.928033 21560 raft_consensus.cc:729] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd4db00a9174eb8b9cf2fb6e1cd392c, State: Initialized, Role: FOLLOWER
I20260925 02:47:36.928704 21560 consensus_queue.cc:260] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:36.929904 21406 heartbeater.cc:503] Master 127.5.56.190:35191 was elected leader, sending a full tablet report...
I20260925 02:47:36.930725 21560 ts_tablet_manager.cc:1428] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Time spent starting tablet: real 0.014s user 0.010s sys 0.006s
I20260925 02:47:36.933169 21561 raft_consensus.cc:348] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:36.933706 21561 raft_consensus.cc:374] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:36.933830 21561 raft_consensus.cc:729] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e899bd70a7f4fa3908a10a38a65e274, State: Initialized, Role: FOLLOWER
I20260925 02:47:36.934702 21561 consensus_queue.cc:260] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:36.936460 21561 ts_tablet_manager.cc:1428] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: Time spent starting tablet: real 0.015s user 0.014s sys 0.000s
I20260925 02:47:36.942711 21568 raft_consensus.cc:482] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:36.942943 21568 raft_consensus.cc:504] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:36.944564 21568 leader_election.cc:288] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cd4db00a9174eb8b9cf2fb6e1cd392c (127.5.56.130:39845), 76fbd1d0e43f4bef871584cb9308a3d7 (127.5.56.131:46183)
W20260925 02:47:36.950836 21271 tablet.cc:2369] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:36.951983 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.945160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:45045 (local address 127.5.56.131:46183)
0925 02:47:36.945362 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:36.945369 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:36.946305 (+ 936us) server_negotiation.cc:407] Connection header received
0925 02:47:36.946566 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:36.946570 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:36.946678 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:36.946801 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:36.947459 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.948197 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.949969 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.950811 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.950996 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.951079 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:36.951684 (+ 605us) server_negotiation.cc:299] Negotiation successful
0925 02:47:36.951795 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 02:47:36.952294 21570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.944742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39845 (local address 127.5.56.129:34903)
0925 02:47:36.946101 (+ 1359us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:36.946118 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:36.946289 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:36.946631 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:36.946635 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:36.946658 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:36.947074 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.947085 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.948171 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.948178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.949742 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.949754 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.951242 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.951247 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.951431 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.951525 (+ 94us) client_negotiation.cc:769] Sending connection context
0925 02:47:36.951624 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 02:47:36.951792 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1242,"mutex_wait_us":92,"thread_start_us":1148,"threads_started":1}
I20260925 02:47:36.952323 21569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.944984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:46183 (local address 127.5.56.129:45045)
0925 02:47:36.946199 (+ 1215us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:36.946213 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:36.946369 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:36.946824 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:36.946828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:36.946856 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:36.947306 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.947316 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.948345 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.948350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.949817 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:36.949829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.951175 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.951186 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:36.951391 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.951521 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 02:47:36.951619 (+ 98us) client_negotiation.cc:240] Negotiation successful
0925 02:47:36.951792 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1112,"mutex_wait_us":111,"thread_start_us":392,"threads_started":1}
I20260925 02:47:36.952636 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.945332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:34903 (local address 127.5.56.130:39845)
0925 02:47:36.945541 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:36.945546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:36.946212 (+ 666us) server_negotiation.cc:407] Connection header received
0925 02:47:36.946440 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:36.946445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:36.946512 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:36.946602 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:36.947215 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.948059 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.949891 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.950916 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.951072 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.951140 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:36.952220 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0925 02:47:36.952343 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:47:36.953377 21496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63ec70fcc4846f989a958175c2c3082" candidate_uuid: "5e899bd70a7f4fa3908a10a38a65e274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" is_pre_election: true
I20260925 02:47:36.953584 21361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63ec70fcc4846f989a958175c2c3082" candidate_uuid: "5e899bd70a7f4fa3908a10a38a65e274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" is_pre_election: true
I20260925 02:47:36.954015 21496 raft_consensus.cc:2454] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e899bd70a7f4fa3908a10a38a65e274 in term 0.
I20260925 02:47:36.954039 21361 raft_consensus.cc:2454] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e899bd70a7f4fa3908a10a38a65e274 in term 0.
I20260925 02:47:36.954763 21159 leader_election.cc:302] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e899bd70a7f4fa3908a10a38a65e274, 8cd4db00a9174eb8b9cf2fb6e1cd392c; no voters:
I20260925 02:47:36.955214 21568 raft_consensus.cc:2781] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:36.955380 21568 raft_consensus.cc:482] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:36.955497 21568 raft_consensus.cc:3037] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:36.957687 21568 raft_consensus.cc:504] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:36.958367 21568 leader_election.cc:288] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [CANDIDATE]: Term 1 election: Requested vote from peers 8cd4db00a9174eb8b9cf2fb6e1cd392c (127.5.56.130:39845), 76fbd1d0e43f4bef871584cb9308a3d7 (127.5.56.131:46183)
I20260925 02:47:36.959017 21496 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63ec70fcc4846f989a958175c2c3082" candidate_uuid: "5e899bd70a7f4fa3908a10a38a65e274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76fbd1d0e43f4bef871584cb9308a3d7"
I20260925 02:47:36.959017 21361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c63ec70fcc4846f989a958175c2c3082" candidate_uuid: "5e899bd70a7f4fa3908a10a38a65e274" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c"
I20260925 02:47:36.959240 21496 raft_consensus.cc:3037] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:36.959239 21361 raft_consensus.cc:3037] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:36.962541 21361 raft_consensus.cc:2454] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e899bd70a7f4fa3908a10a38a65e274 in term 1.
I20260925 02:47:36.962541 21496 raft_consensus.cc:2454] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e899bd70a7f4fa3908a10a38a65e274 in term 1.
I20260925 02:47:36.962985 21157 leader_election.cc:302] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e899bd70a7f4fa3908a10a38a65e274, 76fbd1d0e43f4bef871584cb9308a3d7; no voters:
I20260925 02:47:36.963408 21568 raft_consensus.cc:2781] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:36.963868 21568 raft_consensus.cc:686] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 1 LEADER]: Becoming Leader. State: Replica: 5e899bd70a7f4fa3908a10a38a65e274, State: Running, Role: LEADER
I20260925 02:47:36.964350 21568 consensus_queue.cc:237] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:36.969223 21079 catalog_manager.cc:5495] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e899bd70a7f4fa3908a10a38a65e274 (127.5.56.129). New cstate: current_term: 1 leader_uuid: "5e899bd70a7f4fa3908a10a38a65e274" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } health_report { overall_health: HEALTHY } } }
I20260925 02:47:36.993180 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:36.988191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44372 (local address 127.5.56.129:36505)
0925 02:47:36.988372 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:36.988376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:36.988395 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:47:36.988502 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:36.988506 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:36.988581 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:36.988704 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:36.989059 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.989830 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.991195 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:36.991596 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:36.991698 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:36.992167 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:36.992698 (+ 531us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:36.992813 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:36.992895 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 02:47:36.992964 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260925 02:47:37.001953 21407 tablet.cc:2369] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:37.002116 21361 raft_consensus.cc:1261] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 1 FOLLOWER]: Refusing update from remote peer 5e899bd70a7f4fa3908a10a38a65e274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:37.002111 21496 raft_consensus.cc:1261] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 1 FOLLOWER]: Refusing update from remote peer 5e899bd70a7f4fa3908a10a38a65e274: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:37.003317 21568 consensus_queue.cc:1035] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:37.003690 21572 consensus_queue.cc:1035] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:37.015937 21575 mvcc.cc:205] Tried to move back new op lower bound from 7333087055868764160 to 7333087055729795072. Current Snapshot: MvccSnapshot[applied={T|T < 7333087055868764160}]
I20260925 02:47:37.030947 21080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:37.031252 21080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33910:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:37.035200 21080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Sending DeleteTablet for 3 replicas of tablet c63ec70fcc4846f989a958175c2c3082
I20260925 02:47:37.036393 21476 tablet_service.cc:1514] Processing DeleteTablet for tablet c63ec70fcc4846f989a958175c2c3082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:37 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:56864
I20260925 02:47:37.036645 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 21141
I20260925 02:47:37.036682 21204 tablet_service.cc:1514] Processing DeleteTablet for tablet c63ec70fcc4846f989a958175c2c3082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:37 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:44364
I20260925 02:47:37.036720 21340 tablet_service.cc:1514] Processing DeleteTablet for tablet c63ec70fcc4846f989a958175c2c3082 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:37 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:56868
I20260925 02:47:37.051919 21585 tablet_replica.cc:330] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7: stopping tablet replica
I20260925 02:47:37.052568 21585 raft_consensus.cc:2229] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:37.053268 21585 raft_consensus.cc:2258] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:37.054680 21586 tablet_replica.cc:330] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: stopping tablet replica
I20260925 02:47:37.055136 21586 raft_consensus.cc:2229] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:47:37.055934 21586 raft_consensus.cc:2258] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:37.056559 21587 tablet_replica.cc:330] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: stopping tablet replica
I20260925 02:47:37.056967 21587 raft_consensus.cc:2229] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:37.057467 21587 raft_consensus.cc:2258] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:37.059818 21585 ts_tablet_manager.cc:1905] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 02:47:37 dist-test-slave-n563 krb5kdc[21035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:47:37.065366 21586 ts_tablet_manager.cc:1905] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:37.067273 21585 ts_tablet_manager.cc:1918] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:37.067451 21585 log.cc:1196] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/c63ec70fcc4846f989a958175c2c3082
I20260925 02:47:37.068068 21585 ts_tablet_manager.cc:1939] T c63ec70fcc4846f989a958175c2c3082 P 76fbd1d0e43f4bef871584cb9308a3d7: Deleting consensus metadata
I20260925 02:47:37.069744 21587 ts_tablet_manager.cc:1905] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:37.070261 21064 catalog_manager.cc:4841] TS 76fbd1d0e43f4bef871584cb9308a3d7 (127.5.56.131:46183): tablet c63ec70fcc4846f989a958175c2c3082 (table test-table [id=d778ddb6c6b94b3ca0926f3ce371e3a9]) successfully deleted
I20260925 02:47:37.077186 21586 ts_tablet_manager.cc:1918] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:37.077399 21586 log.cc:1196] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/c63ec70fcc4846f989a958175c2c3082
I20260925 02:47:37.077962 21586 ts_tablet_manager.cc:1939] T c63ec70fcc4846f989a958175c2c3082 P 5e899bd70a7f4fa3908a10a38a65e274: Deleting consensus metadata
I20260925 02:47:37.078037 21558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:37.038035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.5.56.129:36505)
0925 02:47:37.038269 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:37.038277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:37.047708 (+ 9431us) server_negotiation.cc:407] Connection header received
0925 02:47:37.048131 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:37.048140 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:37.048270 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:37.048368 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:37.054089 (+ 5721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:37.055419 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:37.057948 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:37.058910 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:37.059019 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:37.069576 (+ 10557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:37.069596 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:37.069616 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:37.069686 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:37.074084 (+ 4398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:37.075142 (+ 1058us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:37.075158 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:37.075162 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:37.075353 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:37.075871 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:37.075876 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:37.075880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:37.076342 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:37.076623 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:37.077300 (+ 677us) server_negotiation.cc:299] Negotiation successful
0925 02:47:37.077598 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:47:37.079598 21265 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:37.080029 21066 catalog_manager.cc:4841] TS 5e899bd70a7f4fa3908a10a38a65e274 (127.5.56.129:36505): tablet c63ec70fcc4846f989a958175c2c3082 (table test-table [id=d778ddb6c6b94b3ca0926f3ce371e3a9]) successfully deleted
I20260925 02:47:37.080554 21587 ts_tablet_manager.cc:1918] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:37.080734 21587 log.cc:1196] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/c63ec70fcc4846f989a958175c2c3082
I20260925 02:47:37.081288 21587 ts_tablet_manager.cc:1939] T c63ec70fcc4846f989a958175c2c3082 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Deleting consensus metadata
I20260925 02:47:37.082826 21066 catalog_manager.cc:4841] TS 8cd4db00a9174eb8b9cf2fb6e1cd392c (127.5.56.130:39845): tablet c63ec70fcc4846f989a958175c2c3082 (table test-table [id=d778ddb6c6b94b3ca0926f3ce371e3a9]) successfully deleted
I20260925 02:47:37.183583 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 21141
I20260925 02:47:37.203437 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 21276
Sep 25 02:47:37 dist-test-slave-n563 krb5kdc[21035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:47:37.219491 21555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:37.204156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56874 (local address 127.5.56.130:39845)
0925 02:47:37.204336 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:37.204344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:37.204414 (+ 70us) server_negotiation.cc:407] Connection header received
0925 02:47:37.204642 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:37.204646 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:37.204715 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:37.204829 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:37.206125 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:37.206942 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:37.208292 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:37.208673 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:37.208807 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:37.213420 (+ 4613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:37.213435 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:37.213451 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:37.213487 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:37.216421 (+ 2934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:37.217168 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:37.217176 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:37.217178 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:37.217316 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:37.217728 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:37.217735 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:37.217738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:37.218180 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:37.218534 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:37.218890 (+ 356us) server_negotiation.cc:299] Negotiation successful
0925 02:47:37.219120 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:47:37.220211 21401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:37.314709 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 21276
I20260925 02:47:37.333173 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 21412
Sep 25 02:47:37 dist-test-slave-n563 krb5kdc[21035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:47:37.348791 21557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:37.333964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54850 (local address 127.5.56.131:46183)
0925 02:47:37.334158 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:37.334164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:37.334226 (+ 62us) server_negotiation.cc:407] Connection header received
0925 02:47:37.334455 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:37.334464 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:37.334564 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:37.334654 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:37.336020 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:37.336724 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:37.338079 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:37.338668 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:37.338773 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:37.343250 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:37.343268 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:37.343286 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:37.343323 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:37.346027 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:37.346985 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:37.346993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:37.346995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:37.347143 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:37.347479 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:37.347482 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:37.347484 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:37.347843 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:37.348058 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:37.348343 (+ 285us) server_negotiation.cc:299] Negotiation successful
0925 02:47:37.348532 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260925 02:47:37.349434 21536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:37.441545 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 21412
I20260925 02:47:37.459694 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 21048
I20260925 02:47:37.460870 21110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:37.575620 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 21048
I20260925 02:47:57.599557 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:35191
--webserver_interface=127.5.56.190
--webserver_port=38359
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:35191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:47:57.949163 21593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:57.949509 21593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:57.949694 21593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:57.957154 21593 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:47:57.957273 21593 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:47:57.957314 21593 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:47:57.957413 21593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:47:57.957497 21593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:57.957628 21593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:57.957721 21593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:47:57.957806 21593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:47:57.968199 21593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:35191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:35191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=38359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:57.969692 21593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:57.971163 21593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:57.995644 21599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:57.995644 21600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:57.995751 21593 server_base.cc:1034] running on GCE node
W20260925 02:47:57.995645 21602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:57.997104 21593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:57.998879 21593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:58.000144 21593 hybrid_clock.cc:648] HybridClock initialized: now 1790304478000057 us; error 72 us; skew 500 ppm
I20260925 02:47:58.000562 21593 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:58 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:58.004515 21593 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:47:58.006300 21593 webserver.cc:460] Webserver started at http://127.5.56.190:38359/ using document root <none> and password file <none>
I20260925 02:47:58.007186 21593 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:58.007299 21593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:58.013620 21593 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:47:58.016815 21609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:58.018523 21593 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:58.018690 21593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "61af38f9527c4033a0754330e44a9572"
format_stamp: "Formatted at 2026-09-25 02:47:34 on dist-test-slave-n563"
server_key: "2163a5ef27497465f29f7caeffb542e1"
server_key_iv: "768051f70cf397f440c1ac394391754f"
server_key_version: "encryptionkey@0"
I20260925 02:47:58.019855 21593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:58.037560 21593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:58.042337 21593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:58.042686 21593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:58.063083 21593 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:35191
I20260925 02:47:58.063177 21660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:35191 every 8 connection(s)
I20260925 02:47:58.065575 21593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:47:58.066180 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 21593
I20260925 02:47:58.067283 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:36505
--local_ip_for_outbound_sockets=127.5.56.129
--tserver_master_addrs=127.5.56.190:35191
--webserver_port=34599
--webserver_interface=127.5.56.129
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:47:58.073439 21661 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:47:58.077028 21661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Bootstrap starting.
I20260925 02:47:58.107005 21661 log.cc:824] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:58.123183 21661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:47:58.123922 21661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Bootstrap complete.
I20260925 02:47:58.135777 21661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:47:58.136657 21661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61af38f9527c4033a0754330e44a9572, State: Initialized, Role: FOLLOWER
I20260925 02:47:58.137353 21661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:47:58.137609 21661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:47:58.137740 21661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:47:58.137931 21661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 1 FOLLOWER]: Advancing to term 2
I20260925 02:47:58.142298 21661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:47:58.142940 21661 leader_election.cc:302] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61af38f9527c4033a0754330e44a9572; no voters:
I20260925 02:47:58.143612 21661 leader_election.cc:288] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 02:47:58.143853 21665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 02:47:58.145006 21665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 2 LEADER]: Becoming Leader. State: Replica: 61af38f9527c4033a0754330e44a9572, State: Running, Role: LEADER
I20260925 02:47:58.145614 21665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:47:58.146093 21661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:47:58.148561 21667 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61af38f9527c4033a0754330e44a9572. Latest consensus state: current_term: 2 leader_uuid: "61af38f9527c4033a0754330e44a9572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } } }
I20260925 02:47:58.148644 21666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "61af38f9527c4033a0754330e44a9572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } } }
I20260925 02:47:58.149282 21666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:58.149289 21667 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: This master's current role is: LEADER
I20260925 02:47:58.161218 21672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:47:58.174082 21672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:47:58.175704 21672 catalog_manager.cc:1259] Loaded cluster ID: 9d6d6b5d738c423ba1cf6cb9a1064d79
I20260925 02:47:58.175836 21672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:47:58.180539 21672 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:47:58.182264 21672 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Loaded TSK: 0
I20260925 02:47:58.213409 21672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Generated new TSK 1
I20260925 02:47:58.213894 21672 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 02:47:58.479389 21663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:58.479769 21663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:58.479926 21663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:58.487666 21663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:58.487840 21663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:58.488024 21663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:47:58.498482 21663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:36505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=34599
--tserver_master_addrs=127.5.56.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:58.499940 21663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:58.501441 21663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:58.517654 21689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:58.517582 21688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:58.518128 21691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:58.518549 21663 server_base.cc:1034] running on GCE node
I20260925 02:47:58.519506 21663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:58.520887 21663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:58.522176 21663 hybrid_clock.cc:648] HybridClock initialized: now 1790304478522092 us; error 60 us; skew 500 ppm
I20260925 02:47:58.522625 21663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:58 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:58.526897 21663 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:47:58.528826 21663 webserver.cc:460] Webserver started at http://127.5.56.129:34599/ using document root <none> and password file <none>
I20260925 02:47:58.529819 21663 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:58.529986 21663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:58.537181 21663 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260925 02:47:58.540351 21698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:58.541982 21663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:47:58.542166 21663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "5e899bd70a7f4fa3908a10a38a65e274"
format_stamp: "Formatted at 2026-09-25 02:47:35 on dist-test-slave-n563"
server_key: "d904ddf3ed3a1037a4e49ab70156ee7a"
server_key_iv: "7c0456b6efada8451ff44706235889f6"
server_key_version: "encryptionkey@0"
I20260925 02:47:58.543311 21663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:58.556778 21663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:58.561499 21663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:58.561775 21663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:58.563036 21663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:58.564594 21663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:58.564680 21663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:58.564838 21663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:58.564939 21663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:58.600828 21663 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:36505
I20260925 02:47:58.600936 21811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:36505 every 8 connection(s)
I20260925 02:47:58.602491 21663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:47:58.603032 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 21663
I20260925 02:47:58.603878 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:39845
--local_ip_for_outbound_sockets=127.5.56.130
--tserver_master_addrs=127.5.56.190:35191
--webserver_port=39993
--webserver_interface=127.5.56.130
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:47:58 dist-test-slave-n563 krb5kdc[21035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:58.630363 21815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:58.606275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:37447)
0925 02:47:58.607244 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:58.607310 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:58.609093 (+ 1783us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:58.610804 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:58.610813 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:58.611390 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:58.611958 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:58.611976 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:58.614526 (+ 2550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:58.614530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:58.615805 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:58.615817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:58.617336 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:58.617341 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:58.617535 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:58.618280 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:58.618305 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:58.620967 (+ 2662us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:58.625858 (+ 4891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:58.625866 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:58.625879 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:58.626341 (+ 462us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:58.626953 (+ 612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:58.626957 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:58.626959 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:58.627146 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:58.628262 (+ 1116us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:58.628268 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:58.628578 (+ 310us) client_negotiation.cc:769] Sending connection context
0925 02:47:58.628807 (+ 229us) client_negotiation.cc:240] Negotiation successful
0925 02:47:58.629220 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":384,"threads_started":1}
I20260925 02:47:58.631026 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:58.606041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:37447 (local address 127.5.56.190:35191)
0925 02:47:58.606771 (+ 730us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:58.606788 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:58.607459 (+ 671us) server_negotiation.cc:407] Connection header received
0925 02:47:58.609709 (+ 2250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:58.609731 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:58.610222 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:58.610680 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:58.612223 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:58.614294 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:58.615979 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:58.617100 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:58.617313 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:58.621305 (+ 3992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:58.621319 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:58.621342 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:58.621396 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:58.625675 (+ 4279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:58.626542 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:58.626556 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:58.626566 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:58.626821 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:58.627275 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:58.627283 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:58.627288 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:58.628118 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:58.628334 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:58.629107 (+ 773us) server_negotiation.cc:299] Negotiation successful
0925 02:47:58.629608 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":320,"threads_started":1}
I20260925 02:47:58.634115 21812 heartbeater.cc:347] Connected to a master server at 127.5.56.190:35191
I20260925 02:47:58.634472 21812 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:58.635380 21812 heartbeater.cc:511] Master 127.5.56.190:35191 requested a full tablet report, sending...
I20260925 02:47:58.637758 21626 ts_manager.cc:194] Registered new tserver with Master: 5e899bd70a7f4fa3908a10a38a65e274 (127.5.56.129:36505)
I20260925 02:47:58.639348 21626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:37447
W20260925 02:47:58.954953 21816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:58.955324 21816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:58.955494 21816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:58.962965 21816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:58.963140 21816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:58.963331 21816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:47:58.973701 21816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:39845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=39993
--tserver_master_addrs=127.5.56.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:58.975179 21816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:58.976579 21816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:58.990365 21821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:58.990804 21822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:58.990921 21824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:58.991926 21816 server_base.cc:1034] running on GCE node
I20260925 02:47:58.992625 21816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:58.993637 21816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:58.994861 21816 hybrid_clock.cc:648] HybridClock initialized: now 1790304478994797 us; error 60 us; skew 500 ppm
I20260925 02:47:58.995195 21816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:58 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:58.998471 21816 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:47:59.000074 21816 webserver.cc:460] Webserver started at http://127.5.56.130:39993/ using document root <none> and password file <none>
I20260925 02:47:59.000813 21816 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:59.000977 21816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:59.006706 21816 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.002s
I20260925 02:47:59.009596 21831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:59.011104 21816 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:47:59.011291 21816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c"
format_stamp: "Formatted at 2026-09-25 02:47:36 on dist-test-slave-n563"
server_key: "ef27e4405c32059065ab69023f4e92b5"
server_key_iv: "c519912d6ebf4c868706c25226349390"
server_key_version: "encryptionkey@0"
I20260925 02:47:59.012326 21816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:59.049002 21816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:59.053701 21816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:59.054003 21816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:59.055141 21816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:59.056538 21816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:59.056669 21816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:59.056806 21816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:59.056896 21816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:59.092295 21816 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:39845
I20260925 02:47:59.092382 21944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:39845 every 8 connection(s)
I20260925 02:47:59.093865 21816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:47:59.094815 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 21816
I20260925 02:47:59.095578 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:46183
--local_ip_for_outbound_sockets=127.5.56.131
--tserver_master_addrs=127.5.56.190:35191
--webserver_port=33929
--webserver_interface=127.5.56.131
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 02:47:59 dist-test-slave-n563 krb5kdc[21035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304478, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:59.117883 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.098679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:33833 (local address 127.5.56.190:35191)
0925 02:47:59.098905 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.098913 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.098938 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:47:59.100157 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.100166 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.100286 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.100409 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:59.102173 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.103150 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.105786 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.106207 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.106310 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.111211 (+ 4901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:59.111216 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:59.111220 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:59.111259 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:59.113967 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:59.115015 (+ 1048us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:59.115019 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:59.115022 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:59.115145 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:59.115653 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:59.115656 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:59.115658 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:59.116047 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:59.116212 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:59.117283 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0925 02:47:59.117513 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 02:47:59.119741 21947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.097131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:33833)
0925 02:47:59.098029 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:59.098078 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:59.099852 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:59.100639 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:59.100648 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:59.101349 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:59.101977 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.101995 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.103336 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.103344 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.105612 (+ 2268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.105631 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.106505 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.106515 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.106897 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.107734 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:59.107771 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:59.110934 (+ 3163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:59.114135 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:59.114148 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:59.114171 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:59.114892 (+ 721us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:59.115266 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:59.115273 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:59.115280 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:59.115538 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:59.116243 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:59.116255 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:59.116815 (+ 560us) client_negotiation.cc:769] Sending connection context
0925 02:47:59.117174 (+ 359us) client_negotiation.cc:240] Negotiation successful
0925 02:47:59.117875 (+ 701us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":362,"threads_started":1}
I20260925 02:47:59.121577 21945 heartbeater.cc:347] Connected to a master server at 127.5.56.190:35191
I20260925 02:47:59.121958 21945 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:59.123085 21945 heartbeater.cc:511] Master 127.5.56.190:35191 requested a full tablet report, sending...
I20260925 02:47:59.124987 21626 ts_manager.cc:194] Registered new tserver with Master: 8cd4db00a9174eb8b9cf2fb6e1cd392c (127.5.56.130:39845)
I20260925 02:47:59.125838 21626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:33833
W20260925 02:47:59.450103 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:47:59.450484 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:47:59.450655 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:47:59.458213 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:47:59.458343 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:47:59.458557 21948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:47:59.468837 21948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:46183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=33929
--tserver_master_addrs=127.5.56.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:47:59.470306 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:47:59.471750 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:47:59.485425 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:59.485538 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:47:59.485488 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:47:59.491842 21948 server_base.cc:1034] running on GCE node
I20260925 02:47:59.492506 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:47:59.493520 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:47:59.494747 21948 hybrid_clock.cc:648] HybridClock initialized: now 1790304479494677 us; error 66 us; skew 500 ppm
I20260925 02:47:59.495085 21948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:47:59 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:47:59.498185 21948 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:47:59.499874 21948 webserver.cc:460] Webserver started at http://127.5.56.131:33929/ using document root <none> and password file <none>
I20260925 02:47:59.500624 21948 fs_manager.cc:359] Metadata directory not provided
I20260925 02:47:59.500790 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:47:59.506454 21948 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:47:59.509222 21963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:59.510739 21948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:47:59.510947 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "76fbd1d0e43f4bef871584cb9308a3d7"
format_stamp: "Formatted at 2026-09-25 02:47:36 on dist-test-slave-n563"
server_key: "2b3fc311f04964d886286322306f4f0b"
server_key_iv: "da1bdd9104fa8d462947fb2c0d90b1e0"
server_key_version: "encryptionkey@0"
I20260925 02:47:59.512046 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:47:59.524924 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:47:59.529193 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:47:59.529507 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:47:59.530570 21948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:47:59.531926 21948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:47:59.532049 21948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:59.532182 21948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:47:59.532279 21948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:47:59.566172 21948 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:46183
I20260925 02:47:59.566227 22076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:46183 every 8 connection(s)
I20260925 02:47:59.567692 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:47:59.575335 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 21948
Sep 25 02:47:59 dist-test-slave-n563 krb5kdc[21035](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304479, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:47:59.591856 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.570258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:33457 (local address 127.5.56.190:35191)
0925 02:47:59.570503 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.570508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.571734 (+ 1226us) server_negotiation.cc:407] Connection header received
0925 02:47:59.573676 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.573682 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.573780 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.573892 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:59.575817 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.576984 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.579571 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.580351 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.580475 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.584936 (+ 4461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:59.584940 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:59.584943 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:59.584974 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:59.588500 (+ 3526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:59.589322 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:59.589330 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:59.589336 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:59.589492 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:59.589944 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:59.589951 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:59.589958 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:59.590250 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:59.590432 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:59.591183 (+ 751us) server_negotiation.cc:299] Negotiation successful
0925 02:47:59.591428 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":71}
I20260925 02:47:59.592877 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.576475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47922 (local address 127.5.56.190:35191)
0925 02:47:59.577265 (+ 790us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.577269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.577283 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:47:59.577344 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.577347 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.577398 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.577475 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:59.579502 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.580704 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.582758 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.583312 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.583445 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.586623 (+ 3178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:59.586632 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:59.586638 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:59.586725 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:59.589865 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:59.590710 (+ 845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:59.590715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:59.590721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:59.590844 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:59.591257 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:59.591262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:59.591265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:59.591564 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:59.592193 (+ 629us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:59.592347 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 02:47:59.592578 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":686,"thread_start_us":550,"threads_started":1}
I20260925 02:47:59.593288 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.570789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:33457)
0925 02:47:59.571572 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:59.571610 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:59.573396 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:59.574133 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:59.574142 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:59.574752 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:47:59.575628 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.575656 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.577210 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.577217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.579434 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.579451 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.580556 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.580560 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.580790 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.581528 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:47:59.581558 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:47:59.584695 (+ 3137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:47:59.588679 (+ 3984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:59.588687 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:59.588701 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:59.589161 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:59.589608 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:47:59.589611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:47:59.589614 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:47:59.589822 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:47:59.590422 (+ 600us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:47:59.590430 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:47:59.590800 (+ 370us) client_negotiation.cc:769] Sending connection context
0925 02:47:59.591082 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 02:47:59.591659 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":302,"threads_started":1}
I20260925 02:47:59.594833 22077 heartbeater.cc:347] Connected to a master server at 127.5.56.190:35191
I20260925 02:47:59.595147 22077 heartbeater.cc:464] Registering TS with master...
I20260925 02:47:59.596004 22077 heartbeater.cc:511] Master 127.5.56.190:35191 requested a full tablet report, sending...
I20260925 02:47:59.597357 21626 ts_manager.cc:194] Registered new tserver with Master: 76fbd1d0e43f4bef871584cb9308a3d7 (127.5.56.131:46183)
I20260925 02:47:59.598183 21626 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:33457
I20260925 02:47:59.608458 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:47:59.615033 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.609384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.5.56.190:35191)
0925 02:47:59.609631 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.609637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.609994 (+ 357us) server_negotiation.cc:407] Connection header received
0925 02:47:59.610189 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.610193 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.610338 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.610538 (+ 200us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:59.610937 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.611638 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.613020 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.613463 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.613555 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.614051 (+ 496us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:59.614348 (+ 297us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 02:47:59.614734 (+ 386us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47938: token expired
Metrics: {"server-negotiator.queue_time_us":73}
W20260925 02:47:59.615196 22081 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47938: token expired
W20260925 02:47:59.615744 21547 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.5.56.190:35191)
I20260925 02:47:59.629158 22081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.616355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47942 (local address 127.5.56.190:35191)
0925 02:47:59.616711 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.616715 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.616729 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:47:59.616819 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.616824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.616889 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.616965 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:59.618467 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.619228 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.620900 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.621364 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.621453 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.624129 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:59.624133 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:59.624136 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:59.624167 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:59.626920 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:59.627549 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:59.627552 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:59.627554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:59.627667 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:59.627985 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:59.627988 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:59.627990 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:59.628291 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:59.628464 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:59.628714 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 02:47:59.628904 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260925 02:47:59.632748 21626 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47942:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:47:59.641077 21626 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:47:59.643096 21812 heartbeater.cc:503] Master 127.5.56.190:35191 was elected leader, sending a full tablet report...
I20260925 02:47:59.666157 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.653456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39845 (local address 127.0.0.1:38186)
0925 02:47:59.654425 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:59.654475 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:59.654677 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:59.655948 (+ 1271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:59.655960 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:59.656003 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:59.656549 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.656572 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.659410 (+ 2838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.659420 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.662409 (+ 2989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.662425 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.664855 (+ 2430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.664862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.665120 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.665185 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 02:47:59.665311 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 02:47:59.665580 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":778,"threads_started":1}
I20260925 02:47:59.667965 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.653644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38186 (local address 127.5.56.130:39845)
0925 02:47:59.655262 (+ 1618us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.655272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.655299 (+ 27us) server_negotiation.cc:407] Connection header received
0925 02:47:59.655406 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.655416 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.655670 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.656845 (+ 1175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:59.657123 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.659211 (+ 2088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.662664 (+ 3453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.664573 (+ 1909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.666891 (+ 2318us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.667187 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:59.667288 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 02:47:59.667531 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1481,"thread_start_us":1372,"threads_started":1}
I20260925 02:47:59.674068 21880 tablet_service.cc:1467] Processing CreateTablet for tablet 9f511108576b49b295c57b683dd5b597 (DEFAULT_TABLE table=test-table [id=0ab73b48384b49148777c54b1a0b72b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:59.675675 21880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f511108576b49b295c57b683dd5b597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:59.678200 22087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.653506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:46183 (local address 127.0.0.1:39400)
0925 02:47:59.666522 (+ 13016us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:59.666547 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:59.666674 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:59.670580 (+ 3906us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:59.670585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:59.670622 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:59.670867 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.670880 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.673422 (+ 2542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.673428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.675423 (+ 1995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.675440 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.677100 (+ 1660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.677108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.677440 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.677511 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 02:47:59.677589 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 02:47:59.677786 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12832,"mutex_wait_us":165,"spinlock_wait_cycles":3456,"thread_start_us":18236,"threads_started":1}
I20260925 02:47:59.678660 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.655624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43576 (local address 127.5.56.129:36505)
0925 02:47:59.668729 (+ 13105us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.668740 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.669490 (+ 750us) server_negotiation.cc:407] Connection header received
0925 02:47:59.669733 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.669742 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.670037 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.670254 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:59.671430 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.673053 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.675269 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.676671 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.676828 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.677087 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:59.677758 (+ 671us) server_negotiation.cc:299] Negotiation successful
0925 02:47:59.677938 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12891,"thread_start_us":13121,"threads_started":1}
I20260925 02:47:59.678663 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.655216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36505 (local address 127.0.0.1:43576)
0925 02:47:59.669358 (+ 14142us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:59.669372 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:59.669523 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:59.670476 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:59.670483 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:59.670516 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:59.671242 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.671254 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.673233 (+ 1979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.673238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.675109 (+ 1871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.675121 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.676931 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.676937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.677441 (+ 504us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.677505 (+ 64us) client_negotiation.cc:769] Sending connection context
0925 02:47:59.677598 (+ 93us) client_negotiation.cc:240] Negotiation successful
0925 02:47:59.678269 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14023,"mutex_wait_us":755,"thread_start_us":13906,"threads_started":1}
I20260925 02:47:59.679507 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.654091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.5.56.131:46183)
0925 02:47:59.669986 (+ 15895us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.669995 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.670027 (+ 32us) server_negotiation.cc:407] Connection header received
0925 02:47:59.670128 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.670135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.670350 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.670882 (+ 532us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:59.671232 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.673259 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.675583 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.676919 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.678545 (+ 1626us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.678793 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:59.678868 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 02:47:59.679115 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15730,"thread_start_us":19302,"threads_started":1}
I20260925 02:47:59.682564 22011 tablet_service.cc:1467] Processing CreateTablet for tablet 9f511108576b49b295c57b683dd5b597 (DEFAULT_TABLE table=test-table [id=0ab73b48384b49148777c54b1a0b72b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:59.682825 21747 tablet_service.cc:1467] Processing CreateTablet for tablet 9f511108576b49b295c57b683dd5b597 (DEFAULT_TABLE table=test-table [id=0ab73b48384b49148777c54b1a0b72b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:47:59.684032 22011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f511108576b49b295c57b683dd5b597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:59.684520 21747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f511108576b49b295c57b683dd5b597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:47:59.692732 22093 tablet_bootstrap.cc:492] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Bootstrap starting.
I20260925 02:47:59.696739 22093 tablet_bootstrap.cc:654] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:59.697149 22095 tablet_bootstrap.cc:492] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: Bootstrap starting.
I20260925 02:47:59.698256 22093 log.cc:824] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:59.699092 22096 tablet_bootstrap.cc:492] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: Bootstrap starting.
I20260925 02:47:59.700991 22095 tablet_bootstrap.cc:654] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:59.702039 22095 log.cc:824] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:59.703279 22096 tablet_bootstrap.cc:654] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: Neither blocks nor log segments found. Creating new log.
I20260925 02:47:59.704667 22096 log.cc:824] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: Log is configured to *not* fsync() on all Append() calls
I20260925 02:47:59.705240 22093 tablet_bootstrap.cc:492] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: No bootstrap required, opened a new log
I20260925 02:47:59.705457 22095 tablet_bootstrap.cc:492] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: No bootstrap required, opened a new log
I20260925 02:47:59.705564 22093 ts_tablet_manager.cc:1397] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20260925 02:47:59.705765 22095 ts_tablet_manager.cc:1397] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260925 02:47:59.708052 22096 tablet_bootstrap.cc:492] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: No bootstrap required, opened a new log
I20260925 02:47:59.708338 22096 ts_tablet_manager.cc:1397] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260925 02:47:59.713706 22093 raft_consensus.cc:348] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:59.714154 22093 raft_consensus.cc:374] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:59.714257 22093 raft_consensus.cc:729] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cd4db00a9174eb8b9cf2fb6e1cd392c, State: Initialized, Role: FOLLOWER
I20260925 02:47:59.714964 22093 consensus_queue.cc:260] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:59.716226 21945 heartbeater.cc:503] Master 127.5.56.190:35191 was elected leader, sending a full tablet report...
I20260925 02:47:59.716521 22095 raft_consensus.cc:348] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:59.716995 22095 raft_consensus.cc:374] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:59.717061 22093 ts_tablet_manager.cc:1428] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20260925 02:47:59.717092 22095 raft_consensus.cc:729] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76fbd1d0e43f4bef871584cb9308a3d7, State: Initialized, Role: FOLLOWER
I20260925 02:47:59.717686 22095 consensus_queue.cc:260] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:59.718849 22077 heartbeater.cc:503] Master 127.5.56.190:35191 was elected leader, sending a full tablet report...
I20260925 02:47:59.719892 22096 raft_consensus.cc:348] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:59.720333 22096 raft_consensus.cc:374] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:47:59.720428 22096 raft_consensus.cc:729] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e899bd70a7f4fa3908a10a38a65e274, State: Initialized, Role: FOLLOWER
I20260925 02:47:59.720867 22095 ts_tablet_manager.cc:1428] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260925 02:47:59.721180 22096 consensus_queue.cc:260] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:59.723203 22096 ts_tablet_manager.cc:1428] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260925 02:47:59.759555 22101 raft_consensus.cc:482] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:47:59.759929 22101 raft_consensus.cc:504] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:59.761866 22101 leader_election.cc:288] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cd4db00a9174eb8b9cf2fb6e1cd392c (127.5.56.130:39845), 5e899bd70a7f4fa3908a10a38a65e274 (127.5.56.129:36505)
I20260925 02:47:59.768260 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.762228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:43981 (local address 127.5.56.130:39845)
0925 02:47:59.762461 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.762470 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.762492 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:47:59.762570 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.762575 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.762656 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.762775 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:59.763330 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.764348 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.766243 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.767405 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.767520 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.767573 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:59.767962 (+ 389us) server_negotiation.cc:299] Negotiation successful
0925 02:47:59.768060 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:47:59.768386 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.762031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:39845 (local address 127.5.56.131:43981)
0925 02:47:59.762216 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:59.762237 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:59.762356 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:59.762895 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:59.762903 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:59.762945 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:59.763118 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.763127 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.764480 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.764486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.766081 (+ 1595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.766093 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.767571 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.767577 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.767784 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.767825 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:47:59.767935 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:47:59.768078 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":1}
I20260925 02:47:59.769339 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.762587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:50129 (local address 127.5.56.129:36505)
0925 02:47:59.762791 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.762796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.763207 (+ 411us) server_negotiation.cc:407] Connection header received
0925 02:47:59.763363 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.763367 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.763424 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.763490 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:47:59.764095 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.765076 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.767172 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.768312 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.768480 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.768554 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:59.769043 (+ 489us) server_negotiation.cc:299] Negotiation successful
0925 02:47:59.769156 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:47:59.769524 22103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.762426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36505 (local address 127.5.56.131:50129)
0925 02:47:59.763092 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0925 02:47:59.763106 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:47:59.763205 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:47:59.763538 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:47:59.763543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:47:59.763569 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:47:59.763948 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.763960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.765217 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.765221 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.766982 (+ 1761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:47:59.766997 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.768626 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.768632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:47:59.768873 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.768899 (+ 26us) client_negotiation.cc:769] Sending connection context
0925 02:47:59.768974 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:47:59.769118 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"mutex_wait_us":23,"thread_start_us":369,"threads_started":1}
I20260925 02:47:59.769658 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f511108576b49b295c57b683dd5b597" candidate_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" is_pre_election: true
I20260925 02:47:59.770256 21900 raft_consensus.cc:2454] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76fbd1d0e43f4bef871584cb9308a3d7 in term 0.
I20260925 02:47:59.770560 21767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f511108576b49b295c57b683dd5b597" candidate_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e899bd70a7f4fa3908a10a38a65e274" is_pre_election: true
I20260925 02:47:59.771152 21767 raft_consensus.cc:2454] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76fbd1d0e43f4bef871584cb9308a3d7 in term 0.
I20260925 02:47:59.771206 21966 leader_election.cc:302] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76fbd1d0e43f4bef871584cb9308a3d7, 8cd4db00a9174eb8b9cf2fb6e1cd392c; no voters:
I20260925 02:47:59.771656 22101 raft_consensus.cc:2781] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:47:59.771850 22101 raft_consensus.cc:482] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:47:59.771989 22101 raft_consensus.cc:3037] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:59.774232 22101 raft_consensus.cc:504] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:59.774883 22101 leader_election.cc:288] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [CANDIDATE]: Term 1 election: Requested vote from peers 8cd4db00a9174eb8b9cf2fb6e1cd392c (127.5.56.130:39845), 5e899bd70a7f4fa3908a10a38a65e274 (127.5.56.129:36505)
I20260925 02:47:59.775458 21900 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f511108576b49b295c57b683dd5b597" candidate_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c"
I20260925 02:47:59.775514 21767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f511108576b49b295c57b683dd5b597" candidate_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e899bd70a7f4fa3908a10a38a65e274"
I20260925 02:47:59.775701 21900 raft_consensus.cc:3037] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:59.775743 21767 raft_consensus.cc:3037] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:47:59.777980 21900 raft_consensus.cc:2454] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76fbd1d0e43f4bef871584cb9308a3d7 in term 1.
I20260925 02:47:59.777978 21767 raft_consensus.cc:2454] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76fbd1d0e43f4bef871584cb9308a3d7 in term 1.
I20260925 02:47:59.778487 21966 leader_election.cc:302] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76fbd1d0e43f4bef871584cb9308a3d7, 8cd4db00a9174eb8b9cf2fb6e1cd392c; no voters:
I20260925 02:47:59.778846 22101 raft_consensus.cc:2781] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:47:59.779309 22101 raft_consensus.cc:686] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 1 LEADER]: Becoming Leader. State: Replica: 76fbd1d0e43f4bef871584cb9308a3d7, State: Running, Role: LEADER
I20260925 02:47:59.779901 22101 consensus_queue.cc:237] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } }
I20260925 02:47:59.784806 21626 catalog_manager.cc:5495] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 76fbd1d0e43f4bef871584cb9308a3d7 (127.5.56.131). New cstate: current_term: 1 leader_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76fbd1d0e43f4bef871584cb9308a3d7" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 46183 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 } health_report { overall_health: UNKNOWN } } }
W20260925 02:47:59.819275 22078 tablet.cc:2369] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:59.827780 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.822469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39410 (local address 127.5.56.131:46183)
0925 02:47:59.822644 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.822649 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.822669 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:47:59.822892 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.822896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.822980 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.823113 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:47:59.823458 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.824208 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.825550 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.825942 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.826037 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.826570 (+ 533us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:47:59.827191 (+ 621us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:47:59.827309 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:59.827475 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 02:47:59.827545 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:47:59.836768 21900 raft_consensus.cc:1261] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 1 FOLLOWER]: Refusing update from remote peer 76fbd1d0e43f4bef871584cb9308a3d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:59.836768 21767 raft_consensus.cc:1261] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274 [term 1 FOLLOWER]: Refusing update from remote peer 76fbd1d0e43f4bef871584cb9308a3d7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:47:59.837913 22104 consensus_queue.cc:1035] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e899bd70a7f4fa3908a10a38a65e274" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:47:59.838343 22101 consensus_queue.cc:1035] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 39845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 02:47:59.846101 21946 tablet.cc:2369] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:47:59.851384 22107 mvcc.cc:205] Tried to move back new op lower bound from 7333087149399470080 to 7333087149182877696. Current Snapshot: MvccSnapshot[applied={T|T < 7333087149399470080}]
I20260925 02:47:59.852591 22108 mvcc.cc:205] Tried to move back new op lower bound from 7333087149399470080 to 7333087149182877696. Current Snapshot: MvccSnapshot[applied={T|T < 7333087149399470080}]
I20260925 02:47:59.865972 21625 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:59.866290 21625 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47942:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:47:59.870007 21625 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Sending DeleteTablet for 3 replicas of tablet 9f511108576b49b295c57b683dd5b597
I20260925 02:47:59.871336 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f511108576b49b295c57b683dd5b597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:59 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:39400
I20260925 02:47:59.871428 21747 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f511108576b49b295c57b683dd5b597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:59 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:43576
I20260925 02:47:59.871447 21880 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f511108576b49b295c57b683dd5b597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:47:59 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:38186
I20260925 02:47:59.875157 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 21663
I20260925 02:47:59.884416 22119 tablet_replica.cc:330] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: stopping tablet replica
I20260925 02:47:59.884989 22119 raft_consensus.cc:2229] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:59.885722 22119 raft_consensus.cc:2258] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:59.886775 22118 tablet_replica.cc:330] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: stopping tablet replica
I20260925 02:47:59.887246 22118 raft_consensus.cc:2229] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:47:59.888550 22117 tablet_replica.cc:330] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: stopping tablet replica
I20260925 02:47:59.889582 22119 ts_tablet_manager.cc:1905] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:59.887923 22118 raft_consensus.cc:2258] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:59.890515 22117 raft_consensus.cc:2229] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:47:59.891420 22117 raft_consensus.cc:2258] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:47:59.893767 22117 ts_tablet_manager.cc:1905] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:59.896410 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:47:59.876655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43592 (local address 127.5.56.129:36505)
0925 02:47:59.876881 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:47:59.876890 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:47:59.876914 (+ 24us) server_negotiation.cc:407] Connection header received
0925 02:47:59.876994 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:47:59.877003 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:47:59.877141 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:47:59.877290 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:47:59.879726 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.881301 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.883965 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:47:59.884628 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:47:59.884766 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:47:59.889112 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:47:59.889128 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:47:59.889146 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:47:59.889208 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:47:59.893058 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:59.893972 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:59.893981 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:59.893984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:59.894132 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:47:59.894598 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:47:59.894604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:47:59.894606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:47:59.895119 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:47:59.895448 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0925 02:47:59.895760 (+ 312us) server_negotiation.cc:299] Negotiation successful
0925 02:47:59.896003 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 02:47:59.897123 21807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:47:59.899421 22119 ts_tablet_manager.cc:1918] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:59.899602 22119 log.cc:1196] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/9f511108576b49b295c57b683dd5b597
I20260925 02:47:59.900169 22119 ts_tablet_manager.cc:1939] T 9f511108576b49b295c57b683dd5b597 P 8cd4db00a9174eb8b9cf2fb6e1cd392c: Deleting consensus metadata
I20260925 02:47:59.902341 21612 catalog_manager.cc:4841] TS 8cd4db00a9174eb8b9cf2fb6e1cd392c (127.5.56.130:39845): tablet 9f511108576b49b295c57b683dd5b597 (table test-table [id=0ab73b48384b49148777c54b1a0b72b7]) successfully deleted
I20260925 02:47:59.903123 22118 ts_tablet_manager.cc:1905] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:47:59.910964 22118 ts_tablet_manager.cc:1918] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:47:59.911222 22118 log.cc:1196] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/9f511108576b49b295c57b683dd5b597
I20260925 02:47:59.911811 22118 ts_tablet_manager.cc:1939] T 9f511108576b49b295c57b683dd5b597 P 76fbd1d0e43f4bef871584cb9308a3d7: Deleting consensus metadata
I20260925 02:47:59.913899 21610 catalog_manager.cc:4841] TS 76fbd1d0e43f4bef871584cb9308a3d7 (127.5.56.131:46183): tablet 9f511108576b49b295c57b683dd5b597 (table test-table [id=0ab73b48384b49148777c54b1a0b72b7]) successfully deleted
I20260925 02:48:00.023188 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 21663
I20260925 02:48:00.043212 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 21816
W20260925 02:48:00.043365 21612 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.56.129:36505 (error 108)
I20260925 02:48:00.045652 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.045166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36505 (local address 127.0.0.1:43608)
0925 02:48:00.045347 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:00.045472 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:48:00.045943 21612 catalog_manager.cc:4584] TS 5e899bd70a7f4fa3908a10a38a65e274 (127.5.56.129:36505): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f511108576b49b295c57b683dd5b597: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36505: connect: Connection refused (error 111)
I20260925 02:48:00.060325 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.044351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.5.56.130:39845)
0925 02:48:00.044540 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:00.044547 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:00.044647 (+ 100us) server_negotiation.cc:407] Connection header received
0925 02:48:00.044805 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:00.044811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:00.044903 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:00.045027 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:00.046780 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.047966 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.049463 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.050107 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.050238 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:00.053235 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:00.053258 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:00.053280 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:00.053329 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:00.057155 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.057940 (+ 785us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.057950 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.057954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.058120 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.058626 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.058632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.058636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.059149 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:00.059462 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:00.059695 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:48:00.059927 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:48:00.061131 21940 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:00.099658 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.099100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36505 (local address 127.0.0.1:43620)
0925 02:48:00.099356 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:00.099479 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260925 02:48:00.100878 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.100440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36505 (local address 127.0.0.1:43632)
0925 02:48:00.100622 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:00.100711 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:48:00.158821 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 21816
I20260925 02:48:00.161820 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.161200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36505 (local address 127.0.0.1:43640)
0925 02:48:00.161478 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:00.161599 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260925 02:48:00.163143 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.162652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36505 (local address 127.0.0.1:43642)
0925 02:48:00.162884 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:00.162978 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260925 02:48:00.178691 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 21948
I20260925 02:48:00.195648 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.179598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.5.56.131:46183)
0925 02:48:00.179799 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:00.179806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:00.179829 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:48:00.179989 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:00.179996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:00.180111 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:00.180238 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:00.181675 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.182953 (+ 1278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.184293 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:00.184980 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:00.185132 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:00.187935 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:00.187953 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:00.187975 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:00.188033 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:00.192227 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.193079 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.193098 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.193102 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.193307 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:00.193714 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:00.193719 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:00.193723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:00.194284 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:00.194667 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:00.194966 (+ 299us) server_negotiation.cc:299] Negotiation successful
0925 02:48:00.195226 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:48:00.196421 22072 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:00.244560 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.243953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36505 (local address 127.0.0.1:43644)
0925 02:48:00.244212 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:00.244333 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260925 02:48:00.245858 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.245382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36505 (local address 127.0.0.1:43654)
0925 02:48:00.245601 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:00.245709 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260925 02:48:00.324594 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 21948
I20260925 02:48:00.347951 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 21593
I20260925 02:48:00.349152 21656 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:00.472282 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 21593
I20260925 02:48:00.472437 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:00.471584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:36505 (local address 127.0.0.1:43658)
0925 02:48:00.471899 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:00.472130 (+ 231us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:36505: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 02:48:20.495792 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:35191
--webserver_interface=127.5.56.190
--webserver_port=38359
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:35191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:20.837925 22125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:20.838325 22125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:20.838550 22125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:20.846041 22125 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 02:48:20.846206 22125 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 02:48:20.846306 22125 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 02:48:20.846422 22125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:20.846511 22125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:20.846613 22125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:20.846705 22125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:20.846786 22125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:20.857175 22125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:35191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:35191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=38359
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:20.858735 22125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:20.860174 22125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:20.872643 22130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:20.872730 22131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:20.872699 22133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:20.873872 22125 server_base.cc:1034] running on GCE node
I20260925 02:48:20.874931 22125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:20.876529 22125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:20.877825 22125 hybrid_clock.cc:648] HybridClock initialized: now 1790304500877769 us; error 67 us; skew 500 ppm
I20260925 02:48:20.878209 22125 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:20 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304500, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:20.887452 22125 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:48:20.889207 22125 webserver.cc:460] Webserver started at http://127.5.56.190:38359/ using document root <none> and password file <none>
I20260925 02:48:20.890009 22125 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:20.890154 22125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:20.896165 22125 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:48:20.899403 22140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:20.900944 22125 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:48:20.901170 22125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "61af38f9527c4033a0754330e44a9572"
format_stamp: "Formatted at 2026-09-25 02:47:34 on dist-test-slave-n563"
server_key: "2163a5ef27497465f29f7caeffb542e1"
server_key_iv: "768051f70cf397f440c1ac394391754f"
server_key_version: "encryptionkey@0"
I20260925 02:48:20.902268 22125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:20.924525 22125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:20.928758 22125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:20.929076 22125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:20.948779 22125 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:35191
I20260925 02:48:20.948817 22192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:35191 every 8 connection(s)
I20260925 02:48:20.950445 22125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:20.952011 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 22125
I20260925 02:48:20.953042 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:36505
--local_ip_for_outbound_sockets=127.5.56.129
--tserver_master_addrs=127.5.56.190:35191
--webserver_port=34599
--webserver_interface=127.5.56.129
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:48:20.960839 22193 sys_catalog.cc:263] Verifying existing consensus state
I20260925 02:48:20.965109 22193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Bootstrap starting.
I20260925 02:48:20.986588 22193 log.cc:824] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:21.003391 22193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 02:48:21.004055 22193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Bootstrap complete.
I20260925 02:48:21.013617 22193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:48:21.014735 22193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61af38f9527c4033a0754330e44a9572, State: Initialized, Role: FOLLOWER
I20260925 02:48:21.015393 22193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:48:21.015635 22193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:21.015754 22193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:21.015965 22193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 2 FOLLOWER]: Advancing to term 3
I20260925 02:48:21.020159 22193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:48:21.020680 22193 leader_election.cc:302] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 61af38f9527c4033a0754330e44a9572; no voters:
I20260925 02:48:21.021405 22193 leader_election.cc:288] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 02:48:21.021543 22197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 02:48:21.022671 22197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [term 3 LEADER]: Becoming Leader. State: Replica: 61af38f9527c4033a0754330e44a9572, State: Running, Role: LEADER
I20260925 02:48:21.023276 22197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } }
I20260925 02:48:21.023702 22193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:21.026114 22199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 61af38f9527c4033a0754330e44a9572. Latest consensus state: current_term: 3 leader_uuid: "61af38f9527c4033a0754330e44a9572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } } }
I20260925 02:48:21.026590 22199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:21.026559 22198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "61af38f9527c4033a0754330e44a9572" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "61af38f9527c4033a0754330e44a9572" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35191 } } }
I20260925 02:48:21.026906 22198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:21.035334 22202 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:21.045356 22202 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:21.047115 22202 catalog_manager.cc:1259] Loaded cluster ID: 9d6d6b5d738c423ba1cf6cb9a1064d79
I20260925 02:48:21.047333 22202 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:21.051479 22202 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:21.053207 22202 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Loaded TSK: 0
I20260925 02:48:21.053467 22202 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 61af38f9527c4033a0754330e44a9572: Loaded TSK: 1
F20260925 02:48:21.054816 22202 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8ab9abb9ad google::LogMessage::Fail() at ??:0
@ 0x7f8ab9abe88a google::LogMessage::SendToLog() at ??:0
@ 0x7f8ab9abb4de google::LogMessage::Flush() at ??:0
@ 0x7f8ab9abca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8abd81f7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8abd80e73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8acf4ea1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8acf4e9783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8acf5752a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f8acf5750c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f8acf5bf5e9 std::function<>::operator()() at ??:0
@ 0x7f8acf4edcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f8acf4ebc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8acf573610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f8acf573439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8abc707345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8abc70a91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f8abc70a799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f8abc6e5eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8abf9bf6db start_thread at ??:0
@ 0x7f8ab895561f clone at ??:0
W20260925 02:48:21.397362 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:21.397670 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:21.397769 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:21.405299 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:21.405432 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:21.405565 22195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:48:21.415803 22195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:36505
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=34599
--tserver_master_addrs=127.5.56.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:21.417135 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:21.418548 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:21.432371 22240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:21.432402 22241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:21.432371 22243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:21.433524 22195 server_base.cc:1034] running on GCE node
I20260925 02:48:21.434180 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:21.435230 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:21.436448 22195 hybrid_clock.cc:648] HybridClock initialized: now 1790304501436403 us; error 29 us; skew 500 ppm
I20260925 02:48:21.436789 22195 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:21 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304501, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:21.440033 22195 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:48:21.441571 22195 webserver.cc:460] Webserver started at http://127.5.56.129:34599/ using document root <none> and password file <none>
I20260925 02:48:21.442296 22195 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:21.442471 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:21.448028 22195 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260925 02:48:21.451030 22251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:21.452531 22195 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:21.452719 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "5e899bd70a7f4fa3908a10a38a65e274"
format_stamp: "Formatted at 2026-09-25 02:47:35 on dist-test-slave-n563"
server_key: "d904ddf3ed3a1037a4e49ab70156ee7a"
server_key_iv: "7c0456b6efada8451ff44706235889f6"
server_key_version: "encryptionkey@0"
I20260925 02:48:21.454209 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:21.487809 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:21.492443 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:21.492780 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:21.493858 22195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:21.495870 22258 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260925 02:48:21.502019 22258 ts_tablet_manager.cc:1852] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260925 02:48:21.503160 22258 ts_tablet_manager.cc:1905] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:21.511611 22258 ts_tablet_manager.cc:1918] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:21.511869 22258 log.cc:1196] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/9f511108576b49b295c57b683dd5b597
I20260925 02:48:21.512482 22258 ts_tablet_manager.cc:1939] T 9f511108576b49b295c57b683dd5b597 P 5e899bd70a7f4fa3908a10a38a65e274: Deleting consensus metadata
I20260925 02:48:21.512955 22195 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260925 02:48:21.513118 22195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.001s sys 0.000s
I20260925 02:48:21.513286 22195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:21.513386 22195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:21.548615 22195 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:36505
I20260925 02:48:21.548676 22364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:36505 every 8 connection(s)
I20260925 02:48:21.550058 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:21.555179 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 22195
I20260925 02:48:21.555574 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:21.553258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:60373)
0925 02:48:21.554020 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:21.554248 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":288,"threads_started":1}
I20260925 02:48:21.556038 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:39845
--local_ip_for_outbound_sockets=127.5.56.130
--tserver_master_addrs=127.5.56.190:35191
--webserver_port=39993
--webserver_interface=127.5.56.130
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:21.556386 22366 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:35191 (0 consecutive failures): Network error: Failed to ping master at 127.5.56.190:35191: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
I20260925 02:48:21.557402 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:21.556939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:38525)
0925 02:48:21.557141 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:21.557242 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 02:48:21.558503 22368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:21.558046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:52685)
0925 02:48:21.558214 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:21.558312 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:48:21.558768 22366 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:48:21.902483 22369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:21.902880 22369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:21.903066 22369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:21.910529 22369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:21.910728 22369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:21.910943 22369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:48:21.921208 22369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:39845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=39993
--tserver_master_addrs=127.5.56.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:21.922672 22369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:21.924067 22369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:21.938107 22374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:21.938158 22377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:21.938192 22375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:21.939188 22369 server_base.cc:1034] running on GCE node
I20260925 02:48:21.939864 22369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:21.940930 22369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:21.942238 22369 hybrid_clock.cc:648] HybridClock initialized: now 1790304501942164 us; error 59 us; skew 500 ppm
I20260925 02:48:21.942646 22369 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:21 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304501, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:21.945855 22369 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:48:21.947497 22369 webserver.cc:460] Webserver started at http://127.5.56.130:39993/ using document root <none> and password file <none>
I20260925 02:48:21.948233 22369 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:21.948395 22369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:21.954121 22369 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260925 02:48:21.957188 22384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:21.958636 22369 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:21.958828 22369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "8cd4db00a9174eb8b9cf2fb6e1cd392c"
format_stamp: "Formatted at 2026-09-25 02:47:36 on dist-test-slave-n563"
server_key: "ef27e4405c32059065ab69023f4e92b5"
server_key_iv: "c519912d6ebf4c868706c25226349390"
server_key_version: "encryptionkey@0"
I20260925 02:48:21.959843 22369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:21.973476 22369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:21.977548 22369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:21.977849 22369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:21.978941 22369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:21.980324 22369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:21.980454 22369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:21.980584 22369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:21.980679 22369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:22.015053 22369 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:39845
I20260925 02:48:22.015143 22496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:39845 every 8 connection(s)
I20260925 02:48:22.016578 22369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:22.021688 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.019764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:54615)
0925 02:48:22.020530 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.020749 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":281,"threads_started":1}
W20260925 02:48:22.022459 22497 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:35191 (0 consecutive failures): Network error: Failed to ping master at 127.5.56.190:35191: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
I20260925 02:48:22.022935 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 22369
I20260925 02:48:22.023483 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.023132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:59993)
0925 02:48:22.023289 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.023372 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 02:48:22.023617 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:46183
--local_ip_for_outbound_sockets=127.5.56.131
--tserver_master_addrs=127.5.56.190:35191
--webserver_port=33929
--webserver_interface=127.5.56.131
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:36219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:48:22.024389 22499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.024073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:41577)
0925 02:48:22.024202 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.024275 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260925 02:48:22.024626 22497 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 02:48:22.366127 22500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:22.366484 22500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:22.366606 22500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:22.374012 22500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:22.374141 22500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:22.374257 22500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:48:22.384413 22500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:36219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:46183
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=33929
--tserver_master_addrs=127.5.56.190:35191
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:22.385782 22500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:22.387143 22500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:22.400672 22506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:22.400645 22508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:22.400743 22505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:22.401413 22500 server_base.cc:1034] running on GCE node
I20260925 02:48:22.402053 22500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:22.403126 22500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:22.404325 22500 hybrid_clock.cc:648] HybridClock initialized: now 1790304502404255 us; error 63 us; skew 500 ppm
I20260925 02:48:22.404650 22500 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:22 dist-test-slave-n563 krb5kdc[21035](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304502, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:22.407948 22500 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:48:22.409545 22500 webserver.cc:460] Webserver started at http://127.5.56.131:33929/ using document root <none> and password file <none>
I20260925 02:48:22.410274 22500 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:22.410446 22500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:22.416038 22500 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260925 02:48:22.419054 22515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:22.420486 22500 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:22.420675 22500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "76fbd1d0e43f4bef871584cb9308a3d7"
format_stamp: "Formatted at 2026-09-25 02:47:36 on dist-test-slave-n563"
server_key: "2b3fc311f04964d886286322306f4f0b"
server_key_iv: "da1bdd9104fa8d462947fb2c0d90b1e0"
server_key_version: "encryptionkey@0"
I20260925 02:48:22.421702 22500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:22.448923 22500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:22.453586 22500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:22.453888 22500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:22.454965 22500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:22.456362 22500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:22.456508 22500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:22.456641 22500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:22.456736 22500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:22.491369 22500 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:46183
I20260925 02:48:22.491461 22628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:46183 every 8 connection(s)
I20260925 02:48:22.492874 22500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:22.497821 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.495967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:54473)
0925 02:48:22.496709 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.496939 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":288,"threads_started":1}
W20260925 02:48:22.498575 22629 heartbeater.cc:644] Failed to heartbeat to 127.5.56.190:35191 (0 consecutive failures): Network error: Failed to ping master at 127.5.56.190:35191: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
I20260925 02:48:22.499481 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.499111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:37351)
0925 02:48:22.499277 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.499356 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 02:48:22.499928 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 22500
I20260925 02:48:22.500324 22631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.499991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:57681)
0925 02:48:22.500130 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.500209 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 02:48:22.500528 22629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 02:48:22.560175 22633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:22.559366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:35087)
0925 02:48:22.559818 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:22.559932 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20260925 02:48:23.026230 22634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:23.025323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:50425)
0925 02:48:23.025845 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:23.025963 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":239,"threads_started":1}
I20260925 02:48:23.502033 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:23.501229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:60687)
0925 02:48:23.501704 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:23.501795 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":235,"threads_started":1}
I20260925 02:48:23.561848 22636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:23.561137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:37221)
0925 02:48:23.561601 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:23.561690 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":249,"threads_started":1}
I20260925 02:48:24.030706 22637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.027320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:51563)
0925 02:48:24.030301 (+ 2981us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.030458 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2845,"thread_start_us":2683,"threads_started":1}
I20260925 02:48:24.503844 22638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.503101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:58777)
0925 02:48:24.503577 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.503665 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":227,"threads_started":1}
I20260925 02:48:24.563537 22639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:24.562768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:35807)
0925 02:48:24.563214 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:24.563301 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":236,"threads_started":1}
I20260925 02:48:25.032402 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.031632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:43103)
0925 02:48:25.032112 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:25.032199 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":265,"threads_started":1}
I20260925 02:48:25.505558 22641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.504749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:46027)
0925 02:48:25.505254 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:25.505353 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":258,"threads_started":1}
I20260925 02:48:25.565321 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:25.564552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:53247)
0925 02:48:25.565059 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:25.565149 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":272,"threads_started":1}
I20260925 02:48:26.034219 22643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:26.033412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:49613)
0925 02:48:26.033925 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:26.034007 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":276,"threads_started":1}
I20260925 02:48:26.507298 22644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:26.506598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:37625)
0925 02:48:26.507039 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:26.507123 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260925 02:48:26.567118 22645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:26.566332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:57737)
0925 02:48:26.566856 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:26.566941 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":292,"threads_started":1}
I20260925 02:48:27.036074 22646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:27.035251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:47049)
0925 02:48:27.035799 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:27.035892 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":290,"threads_started":1}
I20260925 02:48:27.509262 22647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:27.508388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:40383)
0925 02:48:27.509000 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:27.509086 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":334,"threads_started":1}
I20260925 02:48:27.568885 22648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:27.568101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:56015)
0925 02:48:27.568604 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:27.568688 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":266,"threads_started":1}
I20260925 02:48:28.037837 22649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:28.037067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:34613)
0925 02:48:28.037554 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:28.037652 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":247,"threads_started":1}
I20260925 02:48:28.511096 22650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:28.510294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:55681)
0925 02:48:28.510828 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:28.510925 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":294,"threads_started":1}
I20260925 02:48:28.570814 22651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:28.569886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:49869)
0925 02:48:28.570509 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:28.570604 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":246,"threads_started":1}
I20260925 02:48:29.039870 22652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:29.039010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:48135)
0925 02:48:29.039516 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:29.039603 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":231,"threads_started":1}
I20260925 02:48:29.512840 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:29.512026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:60583)
0925 02:48:29.512544 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:29.512630 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":224,"threads_started":1}
I20260925 02:48:29.572467 22654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:29.571741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:50533)
0925 02:48:29.572208 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:29.572300 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":239,"threads_started":1}
I20260925 02:48:30.041746 22655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:30.040906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:49915)
0925 02:48:30.041475 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:30.041564 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":334,"threads_started":1}
I20260925 02:48:30.514564 22656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:30.513734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:52027)
0925 02:48:30.514261 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:30.514343 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":261,"threads_started":1}
I20260925 02:48:30.574456 22657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:30.573574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:49343)
0925 02:48:30.574081 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:30.574213 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":249,"threads_started":1}
I20260925 02:48:31.043804 22658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:31.042963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:47023)
0925 02:48:31.043524 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:31.043618 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":306,"threads_started":1}
I20260925 02:48:31.516324 22659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:31.515491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:35963)
0925 02:48:31.516043 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:31.516140 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":299,"threads_started":1}
I20260925 02:48:31.576066 22660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:31.575344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:60633)
0925 02:48:31.575817 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:31.575898 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":256,"threads_started":1}
I20260925 02:48:32.045672 22661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:32.044816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:58455)
0925 02:48:32.045351 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:32.045460 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":285,"threads_started":1}
I20260925 02:48:32.518186 22662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:32.517332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:54079)
0925 02:48:32.517888 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:32.517980 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":237,"threads_started":1}
I20260925 02:48:32.577896 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:32.577053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:53421)
0925 02:48:32.577582 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:32.577673 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":275,"threads_started":1}
I20260925 02:48:33.047478 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:33.046741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:50473)
0925 02:48:33.047190 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:33.047291 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":210,"threads_started":1}
I20260925 02:48:33.519987 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:33.519196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:35985)
0925 02:48:33.519699 (+ 503us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:33.519806 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":225,"threads_started":1}
I20260925 02:48:33.579824 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:33.579064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:47675)
0925 02:48:33.579540 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:33.579629 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":232,"threads_started":1}
I20260925 02:48:34.049242 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:34.048471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:36011)
0925 02:48:34.048999 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:34.049077 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":275,"threads_started":1}
I20260925 02:48:34.521680 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:34.520967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:60607)
0925 02:48:34.521428 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:34.521517 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":219,"threads_started":1}
I20260925 02:48:34.581641 22669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:34.580874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:35977)
0925 02:48:34.581355 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:34.581455 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":237,"threads_started":1}
I20260925 02:48:35.051126 22670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:35.050274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:33079)
0925 02:48:35.050835 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:35.050935 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":291,"threads_started":1}
I20260925 02:48:35.524150 22671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:35.522999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:36705)
0925 02:48:35.523690 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:35.523835 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":298,"threads_started":1}
I20260925 02:48:35.583485 22672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:35.582690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:53527)
0925 02:48:35.583202 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:35.583295 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":250,"threads_started":1}
I20260925 02:48:36.052945 22673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:36.052119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:51329)
0925 02:48:36.052664 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:36.052746 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":313,"threads_started":1}
I20260925 02:48:36.526047 22675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:36.525152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:50417)
0925 02:48:36.525649 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:36.525766 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":231,"threads_started":1}
I20260925 02:48:36.585680 22676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:36.584819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:59365)
0925 02:48:36.585393 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:36.585491 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":252,"threads_started":1}
I20260925 02:48:37.054929 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:37.054144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.130:38127)
0925 02:48:37.054615 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:37.054739 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":266,"threads_started":1}
I20260925 02:48:37.505326 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 22195
I20260925 02:48:37.526623 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:37.506707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41870 (local address 127.5.56.129:36505)
0925 02:48:37.507550 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:37.507572 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:37.507627 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:48:37.509448 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:37.509458 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:37.509912 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:37.510283 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:37.511957 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.513842 (+ 1885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.515240 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.515951 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.516162 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:37.519287 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:37.519295 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:37.519313 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:37.519361 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:37.522791 (+ 3430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:37.523648 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:37.523656 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:37.523663 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:37.523846 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:37.524288 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:37.524292 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:37.524294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:37.524938 (+ 644us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:37.525300 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:37.525818 (+ 518us) server_negotiation.cc:299] Negotiation successful
0925 02:48:37.526258 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":457,"threads_started":1}
I20260925 02:48:37.527966 22679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:37.527168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.131:50963)
0925 02:48:37.527683 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:37.527792 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":289,"threads_started":1}
I20260925 02:48:37.528154 22360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:37.616370 22681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:37.615170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35191 (local address 127.5.56.129:35893)
0925 02:48:37.615787 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:37.616027 (+ 240us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.190:35191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":339,"threads_started":1}
I20260925 02:48:37.618582 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 22195
I20260925 02:48:37.637051 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 22369
I20260925 02:48:37.655155 22682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:37.638117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54468 (local address 127.5.56.130:39845)
0925 02:48:37.638772 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:37.638786 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:37.638825 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:48:37.640437 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:37.640449 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:37.640893 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:37.641197 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:37.642736 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.644066 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.645438 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.646128 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.646268 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:37.648987 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:37.648995 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:37.649013 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:37.649054 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:37.652018 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:37.652710 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:37.652717 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:37.652725 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:37.652891 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:37.653214 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:37.653217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:37.653219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:37.653796 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:37.654039 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:37.654452 (+ 413us) server_negotiation.cc:299] Negotiation successful
0925 02:48:37.654862 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":312,"threads_started":1}
I20260925 02:48:37.656575 22492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:37.743966 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 22369
I20260925 02:48:37.762487 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 22500
I20260925 02:48:37.780458 22684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:37.763489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.5.56.131:46183)
0925 02:48:37.764095 (+ 606us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:37.764110 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:37.764154 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:48:37.765812 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:37.765823 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:37.766285 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:37.766662 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:37.767990 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.769230 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.770534 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:37.771233 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:37.771399 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:37.774116 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:37.774124 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:37.774141 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:37.774198 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:37.777243 (+ 3045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:37.777973 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:37.777980 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:37.777989 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:37.778169 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:37.778621 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:37.778624 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:37.778626 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:37.779191 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:37.779432 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:37.779782 (+ 350us) server_negotiation.cc:299] Negotiation successful
0925 02:48:37.780138 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":293,"threads_started":1}
I20260925 02:48:37.781876 22624 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:37.866677 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 22500
2026-09-25T02:48:37Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64665 ms)
[----------] 30 tests from SecurityITest (305048 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:37 dist-test-slave-n563 krb5kdc[22688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:37 dist-test-slave-n563 krb5kdc[22688](info): set up 2 sockets
Sep 25 02:48:37 dist-test-slave-n563 krb5kdc[22688](info): commencing operation
krb5kdc: starting...
W20260925 02:48:38.976666 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:48:39 dist-test-slave-n563 krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:48:39.246888 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:44499
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:44499
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:39.593664 22701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:39.593990 22701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:39.594082 22701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:39.601569 22701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:39.601702 22701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:39.601751 22701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:39.601804 22701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:39.601850 22701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:39.612249 22701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:44499
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:44499
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:39.613646 22701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:39.615007 22701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:39.627655 22707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:39.627678 22709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:39.627864 22706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:39.628494 22701 server_base.cc:1034] running on GCE node
I20260925 02:48:39.629371 22701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:39.631008 22701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:39.632252 22701 hybrid_clock.cc:648] HybridClock initialized: now 1790304519632211 us; error 45 us; skew 500 ppm
I20260925 02:48:39.632642 22701 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:39 dist-test-slave-n563 krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304519, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:39.635993 22701 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:48:39.637655 22701 webserver.cc:460] Webserver started at http://127.5.56.190:42417/ using document root <none> and password file <none>
I20260925 02:48:39.638469 22701 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:39.638622 22701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:39.638950 22701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:39.642377 22701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "40312d6498f4495696de0eec5320f676"
format_stamp: "Formatted at 2026-09-25 02:48:39 on dist-test-slave-n563"
server_key: "824e142055a3936f3f9bbb46b6cabf9c"
server_key_iv: "4b7408a8528b6cade59c3b1c44aa9ae9"
server_key_version: "encryptionkey@0"
I20260925 02:48:39.643280 22701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "40312d6498f4495696de0eec5320f676"
format_stamp: "Formatted at 2026-09-25 02:48:39 on dist-test-slave-n563"
server_key: "824e142055a3936f3f9bbb46b6cabf9c"
server_key_iv: "4b7408a8528b6cade59c3b1c44aa9ae9"
server_key_version: "encryptionkey@0"
I20260925 02:48:39.649158 22701 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:48:39.652908 22716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:39.654285 22701 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:48:39.654521 22701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "40312d6498f4495696de0eec5320f676"
format_stamp: "Formatted at 2026-09-25 02:48:39 on dist-test-slave-n563"
server_key: "824e142055a3936f3f9bbb46b6cabf9c"
server_key_iv: "4b7408a8528b6cade59c3b1c44aa9ae9"
server_key_version: "encryptionkey@0"
I20260925 02:48:39.654739 22701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:39.695770 22701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:39.700050 22701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:39.700376 22701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:39.719916 22701 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:44499
I20260925 02:48:39.719939 22767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:44499 every 8 connection(s)
I20260925 02:48:39.721493 22701 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:39.723556 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 22701
I20260925 02:48:39.723773 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:48:39.724459 5346 external_mini_cluster.cc:1428] Setting key a8643e0a7f89b94515b1916c9ce095b6
I20260925 02:48:39.725898 22768 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:48:39 dist-test-slave-n563 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:39.740273 22768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676: Bootstrap starting.
I20260925 02:48:39.744593 22768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:39.745910 22768 log.cc:824] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:39.749586 22768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676: No bootstrap required, opened a new log
I20260925 02:48:39.750156 22771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:39.726594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42782 (local address 127.5.56.190:44499)
0925 02:48:39.727457 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:39.727473 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:39.727523 (+ 50us) server_negotiation.cc:407] Connection header received
0925 02:48:39.729554 (+ 2031us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:39.729584 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:39.730128 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:39.730629 (+ 501us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:39.732159 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.734252 (+ 2093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.735683 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:39.736742 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:39.736937 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:39.741206 (+ 4269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:39.741217 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:39.741242 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:39.741295 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:39.745307 (+ 4012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:39.746068 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:39.746078 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:39.746088 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:39.746296 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:39.746842 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:39.746845 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:39.746847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:39.747588 (+ 741us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:39.747904 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:39.748275 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 02:48:39.748776 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":613,"thread_start_us":339,"threads_started":1}
I20260925 02:48:39.758018 22768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40312d6498f4495696de0eec5320f676" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44499 } }
I20260925 02:48:39.758499 22768 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:39.758625 22768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40312d6498f4495696de0eec5320f676, State: Initialized, Role: FOLLOWER
I20260925 02:48:39.759328 22768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40312d6498f4495696de0eec5320f676" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44499 } }
I20260925 02:48:39.759593 22768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:39.759795 22768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:39.760022 22768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:39.762292 22768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40312d6498f4495696de0eec5320f676" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44499 } }
I20260925 02:48:39.762881 22768 leader_election.cc:302] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40312d6498f4495696de0eec5320f676; no voters:
I20260925 02:48:39.763568 22768 leader_election.cc:288] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:39.763713 22773 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:39.764420 22773 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [term 1 LEADER]: Becoming Leader. State: Replica: 40312d6498f4495696de0eec5320f676, State: Running, Role: LEADER
I20260925 02:48:39.765055 22773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40312d6498f4495696de0eec5320f676" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44499 } }
I20260925 02:48:39.765918 22768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:39.768626 22775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40312d6498f4495696de0eec5320f676. Latest consensus state: current_term: 1 leader_uuid: "40312d6498f4495696de0eec5320f676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40312d6498f4495696de0eec5320f676" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44499 } } }
I20260925 02:48:39.769066 22775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:39.769012 22774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40312d6498f4495696de0eec5320f676" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40312d6498f4495696de0eec5320f676" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 44499 } } }
I20260925 02:48:39.769290 22774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:39.771764 22782 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:39.776054 22782 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:39.785817 22782 catalog_manager.cc:1347] Generated new cluster ID: cf1a2a92ca4d4fa589516aad6b6be04a
I20260925 02:48:39.786029 22782 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:39.813468 22782 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:39.814987 22782 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:39.820812 22782 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676: Generated new TSK 0
I20260925 02:48:39.821759 22782 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:48:39.902323 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:44499
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:40.254035 22794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:40.254423 22794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:40.254602 22794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:40.262513 22794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:40.262681 22794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:40.262873 22794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:48:40.273505 22794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:44499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:40.274996 22794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:40.276340 22794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:40.289994 22799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:40.290030 22800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:40.290067 22802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:40.290978 22794 server_base.cc:1034] running on GCE node
I20260925 02:48:40.291625 22794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:40.292618 22794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:40.293856 22794 hybrid_clock.cc:648] HybridClock initialized: now 1790304520293792 us; error 55 us; skew 500 ppm
I20260925 02:48:40.294190 22794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:40 dist-test-slave-n563 krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304520, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:40.297513 22794 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:48:40.299162 22794 webserver.cc:460] Webserver started at http://127.5.56.129:42675/ using document root <none> and password file <none>
I20260925 02:48:40.299882 22794 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:40.300034 22794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:40.300375 22794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:40.303561 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "c6d942b1e58f47c1881cdd7a6dde18cc"
format_stamp: "Formatted at 2026-09-25 02:48:40 on dist-test-slave-n563"
server_key: "5494b58801593837796b3a4bd502f57f"
server_key_iv: "b0cdbc5f18a6662e70863a7e5a91b21a"
server_key_version: "encryptionkey@0"
I20260925 02:48:40.304381 22794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "c6d942b1e58f47c1881cdd7a6dde18cc"
format_stamp: "Formatted at 2026-09-25 02:48:40 on dist-test-slave-n563"
server_key: "5494b58801593837796b3a4bd502f57f"
server_key_iv: "b0cdbc5f18a6662e70863a7e5a91b21a"
server_key_version: "encryptionkey@0"
I20260925 02:48:40.309809 22794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:48:40.313692 22809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:40.315164 22794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:40.315368 22794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "c6d942b1e58f47c1881cdd7a6dde18cc"
format_stamp: "Formatted at 2026-09-25 02:48:40 on dist-test-slave-n563"
server_key: "5494b58801593837796b3a4bd502f57f"
server_key_iv: "b0cdbc5f18a6662e70863a7e5a91b21a"
server_key_version: "encryptionkey@0"
I20260925 02:48:40.315590 22794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:40.335336 22794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:40.339920 22794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:40.340229 22794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:40.341331 22794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:40.342816 22794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:40.342921 22794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:40.343079 22794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:40.343153 22794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:40.378639 22794 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:35977
I20260925 02:48:40.378684 22921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:35977 every 8 connection(s)
I20260925 02:48:40.379987 22794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:40.382364 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 22794
I20260925 02:48:40.382673 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:40.383431 5346 external_mini_cluster.cc:1428] Setting key 7ebe9fa22b73121d53411061ff28df55
Sep 25 02:48:40 dist-test-slave-n563 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304520, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:40.404455 22924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:40.382665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:43045 (local address 127.5.56.190:44499)
0925 02:48:40.383169 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:40.383176 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:40.384376 (+ 1200us) server_negotiation.cc:407] Connection header received
0925 02:48:40.386542 (+ 2166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:40.386546 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:40.386611 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:40.386700 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:40.388881 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.389613 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.392165 (+ 2552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.392562 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.392663 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:40.397210 (+ 4547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:40.397219 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:40.397227 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:40.397275 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:40.400932 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:40.401729 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:40.401737 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:40.401743 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:40.401900 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:40.402343 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:40.402350 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:40.402357 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:40.402779 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:40.402931 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:40.403665 (+ 734us) server_negotiation.cc:299] Negotiation successful
0925 02:48:40.403950 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":242,"threads_started":1}
I20260925 02:48:40.405093 22925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:40.383323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:44499 (local address 127.5.56.129:43045)
0925 02:48:40.384201 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:40.384253 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:40.386220 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:40.386992 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:40.387007 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:40.387865 (+ 858us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:40.388701 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:40.388728 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.389787 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.389795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:40.392020 (+ 2225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:40.392038 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.392811 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.392820 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:40.393155 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:40.393933 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:40.393974 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:40.396954 (+ 2980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:40.401110 (+ 4156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:40.401117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:40.401133 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:40.401586 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:40.402007 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:40.402010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:40.402012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:40.402193 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:40.402901 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:40.402907 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:40.403256 (+ 349us) client_negotiation.cc:769] Sending connection context
0925 02:48:40.403496 (+ 240us) client_negotiation.cc:240] Negotiation successful
0925 02:48:40.403913 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":590,"thread_start_us":358,"threads_started":1}
I20260925 02:48:40.406844 22922 heartbeater.cc:347] Connected to a master server at 127.5.56.190:44499
I20260925 02:48:40.407167 22922 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:40.407927 22922 heartbeater.cc:511] Master 127.5.56.190:44499 requested a full tablet report, sending...
I20260925 02:48:40.410178 22733 ts_manager.cc:194] Registered new tserver with Master: c6d942b1e58f47c1881cdd7a6dde18cc (127.5.56.129:35977)
I20260925 02:48:40.411698 22733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:43045
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:48:40.445242 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:44499
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:40.784205 22928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:40.784555 22928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:40.784735 22928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:40.792191 22928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:40.792358 22928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:40.792546 22928 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:48:40.802745 22928 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:44499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:40.804137 22928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:40.805497 22928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:40.819571 22934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:40.819649 22936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:40.820150 22933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:40.820835 22928 server_base.cc:1034] running on GCE node
I20260925 02:48:40.821493 22928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:40.822562 22928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:40.823799 22928 hybrid_clock.cc:648] HybridClock initialized: now 1790304520823743 us; error 49 us; skew 500 ppm
I20260925 02:48:40.824142 22928 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:40 dist-test-slave-n563 krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304520, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:40.833246 22928 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:48:40.834990 22928 webserver.cc:460] Webserver started at http://127.5.56.130:38719/ using document root <none> and password file <none>
I20260925 02:48:40.835693 22928 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:40.835798 22928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:40.836066 22928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:40.839133 22928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "bb6181514be64df7b197c533d7e56804"
format_stamp: "Formatted at 2026-09-25 02:48:40 on dist-test-slave-n563"
server_key: "2ed57ceb141481be4fbe6271eccf3913"
server_key_iv: "dc33387c22ebda7c6945b05a72447818"
server_key_version: "encryptionkey@0"
I20260925 02:48:40.839902 22928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "bb6181514be64df7b197c533d7e56804"
format_stamp: "Formatted at 2026-09-25 02:48:40 on dist-test-slave-n563"
server_key: "2ed57ceb141481be4fbe6271eccf3913"
server_key_iv: "dc33387c22ebda7c6945b05a72447818"
server_key_version: "encryptionkey@0"
I20260925 02:48:40.845402 22928 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 02:48:40.848987 22943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:40.850452 22928 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:48:40.850631 22928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "bb6181514be64df7b197c533d7e56804"
format_stamp: "Formatted at 2026-09-25 02:48:40 on dist-test-slave-n563"
server_key: "2ed57ceb141481be4fbe6271eccf3913"
server_key_iv: "dc33387c22ebda7c6945b05a72447818"
server_key_version: "encryptionkey@0"
I20260925 02:48:40.850881 22928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:40.870253 22928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:40.874357 22928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:40.874744 22928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:40.875785 22928 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:40.877211 22928 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:40.877305 22928 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:40.877489 22928 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:40.877543 22928 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:40.912904 22928 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:40801
I20260925 02:48:40.912995 23058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:40801 every 8 connection(s)
I20260925 02:48:40.914275 22928 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:40.921478 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 22928
I20260925 02:48:40.921674 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:40.922219 5346 external_mini_cluster.cc:1428] Setting key 04ff56c13e3eab946594485bc6e51339
Sep 25 02:48:40 dist-test-slave-n563 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304520, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:40.936815 23061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:40.916883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:42411 (local address 127.5.56.190:44499)
0925 02:48:40.917308 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:40.917313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:40.918348 (+ 1035us) server_negotiation.cc:407] Connection header received
0925 02:48:40.920403 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:40.920409 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:40.920529 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:40.920659 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:40.922926 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.923809 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.926101 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.926596 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.926704 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:40.931024 (+ 4320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:40.931030 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:40.931033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:40.931069 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:40.933656 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:40.934458 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:40.934462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:40.934464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:40.934590 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:40.935041 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:40.935046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:40.935048 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:40.935322 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:40.935485 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:40.936193 (+ 708us) server_negotiation.cc:299] Negotiation successful
0925 02:48:40.936423 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":209,"threads_started":1}
I20260925 02:48:40.938148 23062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:40.917492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:44499 (local address 127.5.56.130:42411)
0925 02:48:40.918185 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:40.918230 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:40.920130 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:40.920911 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:40.920919 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:40.921768 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:40.922643 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:40.922672 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.923976 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.923981 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:40.925933 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:40.925953 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:40.926841 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:40.926851 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:40.927193 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:40.927951 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:40.927990 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:40.930765 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:40.933823 (+ 3058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:40.933833 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:40.933851 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:40.934293 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:40.934707 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:40.934714 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:40.934718 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:40.934913 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:40.935457 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:40.935466 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:40.935828 (+ 362us) client_negotiation.cc:769] Sending connection context
0925 02:48:40.936094 (+ 266us) client_negotiation.cc:240] Negotiation successful
0925 02:48:40.936662 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":287,"threads_started":1}
I20260925 02:48:40.939807 23059 heartbeater.cc:347] Connected to a master server at 127.5.56.190:44499
I20260925 02:48:40.940158 23059 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:40.941062 23059 heartbeater.cc:511] Master 127.5.56.190:44499 requested a full tablet report, sending...
I20260925 02:48:40.942772 22733 ts_manager.cc:194] Registered new tserver with Master: bb6181514be64df7b197c533d7e56804 (127.5.56.130:40801)
I20260925 02:48:40.943642 22733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:42411
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:48:40.982889 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:44499
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:38697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:41.340634 23065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:41.341020 23065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:41.341221 23065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:41.349282 23065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:41.349491 23065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:41.349689 23065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:48:41.362136 23065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:38697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:44499
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:41.363727 23065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:41.365197 23065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:41.380189 23071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:41.380167 23070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:41.382741 23073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:41.386601 23065 server_base.cc:1034] running on GCE node
I20260925 02:48:41.387288 23065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:41.388373 23065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:41.389582 23065 hybrid_clock.cc:648] HybridClock initialized: now 1790304521389535 us; error 41 us; skew 500 ppm
I20260925 02:48:41.389920 23065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:41 dist-test-slave-n563 krb5kdc[22688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:41.393144 23065 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:48:41.394769 23065 webserver.cc:460] Webserver started at http://127.5.56.131:41429/ using document root <none> and password file <none>
I20260925 02:48:41.395511 23065 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:41.395666 23065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:41.396001 23065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:41.399115 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "bacfe83465f747e08271dc497140af6b"
format_stamp: "Formatted at 2026-09-25 02:48:41 on dist-test-slave-n563"
server_key: "bd2d33e32284ddb55472c76540e85d90"
server_key_iv: "183ccdf9a43cc2321ba31c6a49cb1c33"
server_key_version: "encryptionkey@0"
I20260925 02:48:41.399936 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "bacfe83465f747e08271dc497140af6b"
format_stamp: "Formatted at 2026-09-25 02:48:41 on dist-test-slave-n563"
server_key: "bd2d33e32284ddb55472c76540e85d90"
server_key_iv: "183ccdf9a43cc2321ba31c6a49cb1c33"
server_key_version: "encryptionkey@0"
I20260925 02:48:41.405333 23065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:48:41.409075 23080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:41.410605 23065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:41.410789 23065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "bacfe83465f747e08271dc497140af6b"
format_stamp: "Formatted at 2026-09-25 02:48:41 on dist-test-slave-n563"
server_key: "bd2d33e32284ddb55472c76540e85d90"
server_key_iv: "183ccdf9a43cc2321ba31c6a49cb1c33"
server_key_version: "encryptionkey@0"
I20260925 02:48:41.411017 23065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:41.414796 22922 heartbeater.cc:503] Master 127.5.56.190:44499 was elected leader, sending a full tablet report...
I20260925 02:48:41.423154 23065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:41.427544 23065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:41.427855 23065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:41.428843 23065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:41.430461 23065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:41.430562 23065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260925 02:48:41.430719 23065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:41.430792 23065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:41.464965 23065 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:38691
I20260925 02:48:41.465013 23193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:38691 every 8 connection(s)
I20260925 02:48:41.466495 23065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:41.469574 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 23065
I20260925 02:48:41.469769 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:41.470431 5346 external_mini_cluster.cc:1428] Setting key 970719c908aef79f7e58ed4f6ac277ba
Sep 25 02:48:41 dist-test-slave-n563 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304521, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:41.487792 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.469189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:60445 (local address 127.5.56.190:44499)
0925 02:48:41.469738 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.469743 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.470960 (+ 1217us) server_negotiation.cc:407] Connection header received
0925 02:48:41.473120 (+ 2160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.473125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.473217 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.473344 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:41.475018 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.475761 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.477304 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.477700 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.477793 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.481410 (+ 3617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:41.481415 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:41.481418 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:41.481453 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:41.484197 (+ 2744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:41.485178 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:41.485184 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:41.485189 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:41.485327 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:41.485861 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:41.485865 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:41.485867 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:41.486220 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:41.486440 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:41.487156 (+ 716us) server_negotiation.cc:299] Negotiation successful
0925 02:48:41.487394 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":263,"threads_started":1}
I20260925 02:48:41.488843 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.469924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:44499 (local address 127.5.56.131:60445)
0925 02:48:41.470788 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:41.470835 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:41.472864 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:41.473628 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:41.473643 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:41.474234 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:41.474872 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.474890 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.475918 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.475922 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.477175 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.477187 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.477884 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.477888 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.478082 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.478801 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:41.478823 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:41.481161 (+ 2338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:41.484403 (+ 3242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:41.484413 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:41.484432 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:41.485057 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:41.485466 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:41.485469 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:41.485472 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:41.485740 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:41.486372 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:41.486400 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:41.486773 (+ 373us) client_negotiation.cc:769] Sending connection context
0925 02:48:41.487048 (+ 275us) client_negotiation.cc:240] Negotiation successful
0925 02:48:41.487635 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":382,"threads_started":1}
I20260925 02:48:41.490206 23194 heartbeater.cc:347] Connected to a master server at 127.5.56.190:44499
I20260925 02:48:41.490590 23194 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:41.491331 23194 heartbeater.cc:511] Master 127.5.56.190:44499 requested a full tablet report, sending...
I20260925 02:48:41.492776 22733 ts_manager.cc:194] Registered new tserver with Master: bacfe83465f747e08271dc497140af6b (127.5.56.131:38691)
I20260925 02:48:41.493646 22733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:60445
I20260925 02:48:41.498667 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:48:41.516238 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.503382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42798 (local address 127.5.56.190:44499)
0925 02:48:41.503554 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.503559 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.504126 (+ 567us) server_negotiation.cc:407] Connection header received
0925 02:48:41.504269 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.504273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.504343 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.504427 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:41.505779 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.506629 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.507974 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.508359 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.508448 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.511062 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:41.511067 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:41.511071 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:41.511103 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:41.513547 (+ 2444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:41.514176 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:41.514179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:41.514182 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:41.514371 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:41.514772 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:41.514775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:41.514777 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:41.515109 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:41.515301 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:41.515639 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 02:48:41.515882 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:48:41.520792 22733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42798:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:41.527717 22733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:41.552520 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.540277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:40801 (local address 127.0.0.1:38878)
0925 02:48:41.542034 (+ 1757us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:41.542078 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:41.542247 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:41.545822 (+ 3575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:41.545826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:41.545850 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:41.546177 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.546186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.547703 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.547708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.549667 (+ 1959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.549678 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.551405 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.551412 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.551599 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.551658 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 02:48:41.551807 (+ 149us) client_negotiation.cc:240] Negotiation successful
0925 02:48:41.552024 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1507,"mutex_wait_us":61,"thread_start_us":1350,"threads_started":1}
I20260925 02:48:41.553179 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.543038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:38691 (local address 127.0.0.1:38216)
0925 02:48:41.543731 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:41.543754 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:41.543937 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:41.545258 (+ 1321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:41.545266 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:41.545292 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:41.545752 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.545768 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.548031 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.548037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.550444 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.550456 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.551946 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.551952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.552105 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.552137 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:48:41.552214 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 02:48:41.552335 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":81,"spinlock_wait_cycles":2816,"thread_start_us":740,"threads_started":1}
I20260925 02:48:41.553457 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.543636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.5.56.131:38691)
0925 02:48:41.544590 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.544602 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.544640 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:48:41.544756 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.544766 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.545029 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.545969 (+ 940us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:41.546281 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.547873 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.550609 (+ 2736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.551697 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.552624 (+ 927us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.552877 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:41.552955 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 02:48:41.553128 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":739,"thread_start_us":1038,"threads_started":1}
I20260925 02:48:41.553594 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.540475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.5.56.130:40801)
0925 02:48:41.545337 (+ 4862us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.545347 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.545375 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:48:41.545471 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.545480 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.545642 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.545835 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:41.546342 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.547540 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.549839 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.551165 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.551320 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.551527 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:41.553082 (+ 1555us) server_negotiation.cc:299] Negotiation successful
0925 02:48:41.553248 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4700,"thread_start_us":4586,"threads_started":1}
I20260925 02:48:41.553603 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.540551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:35977 (local address 127.0.0.1:35230)
0925 02:48:41.543342 (+ 2791us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:41.543360 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:41.543915 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:41.544883 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:41.544891 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:41.544918 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:41.545289 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.545308 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.547451 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.547456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.550203 (+ 2747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.550219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.552722 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.552731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.552911 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.552944 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 02:48:41.553038 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 02:48:41.553192 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2669,"mutex_wait_us":69,"spinlock_wait_cycles":4992,"thread_start_us":2508,"threads_started":1}
I20260925 02:48:41.554237 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.541250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35230 (local address 127.5.56.129:35977)
0925 02:48:41.542912 (+ 1662us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.542921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.543809 (+ 888us) server_negotiation.cc:407] Connection header received
0925 02:48:41.544122 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.544132 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.544405 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.544595 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:41.545534 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.547276 (+ 1742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.550379 (+ 3103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.552212 (+ 1833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.552340 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.552847 (+ 507us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:41.553748 (+ 901us) server_negotiation.cc:299] Negotiation successful
0925 02:48:41.553932 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1474,"thread_start_us":1317,"threads_started":1}
I20260925 02:48:41.557601 22857 tablet_service.cc:1467] Processing CreateTablet for tablet a4701692bd7c4ba7ad0637baec4f5e41 (DEFAULT_TABLE table=test-table [id=f7b23fcefbc344efa97fec61238af37d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:41.557631 22993 tablet_service.cc:1467] Processing CreateTablet for tablet a4701692bd7c4ba7ad0637baec4f5e41 (DEFAULT_TABLE table=test-table [id=f7b23fcefbc344efa97fec61238af37d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:41.557641 23129 tablet_service.cc:1467] Processing CreateTablet for tablet a4701692bd7c4ba7ad0637baec4f5e41 (DEFAULT_TABLE table=test-table [id=f7b23fcefbc344efa97fec61238af37d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:41.559073 22993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4701692bd7c4ba7ad0637baec4f5e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:41.559072 23129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4701692bd7c4ba7ad0637baec4f5e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:41.559072 22857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4701692bd7c4ba7ad0637baec4f5e41. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:41.571022 23213 tablet_bootstrap.cc:492] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: Bootstrap starting.
I20260925 02:48:41.575196 23215 tablet_bootstrap.cc:492] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: Bootstrap starting.
I20260925 02:48:41.576087 23213 tablet_bootstrap.cc:654] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:41.577028 23214 tablet_bootstrap.cc:492] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: Bootstrap starting.
I20260925 02:48:41.577504 23213 log.cc:824] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:41.579166 23215 tablet_bootstrap.cc:654] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:41.580332 23214 tablet_bootstrap.cc:654] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:41.580502 23215 log.cc:824] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:41.581084 23213 tablet_bootstrap.cc:492] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: No bootstrap required, opened a new log
I20260925 02:48:41.581441 23213 ts_tablet_manager.cc:1397] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260925 02:48:41.581648 23214 log.cc:824] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:41.584705 23215 tablet_bootstrap.cc:492] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: No bootstrap required, opened a new log
I20260925 02:48:41.584976 23214 tablet_bootstrap.cc:492] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: No bootstrap required, opened a new log
I20260925 02:48:41.585001 23215 ts_tablet_manager.cc:1397] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260925 02:48:41.585242 23214 ts_tablet_manager.cc:1397] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260925 02:48:41.592265 23215 raft_consensus.cc:348] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6181514be64df7b197c533d7e56804" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 40801 } } peers { permanent_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35977 } } peers { permanent_uuid: "bacfe83465f747e08271dc497140af6b" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38691 } }
I20260925 02:48:41.592479 23213 raft_consensus.cc:348] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6181514be64df7b197c533d7e56804" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 40801 } } peers { permanent_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35977 } } peers { permanent_uuid: "bacfe83465f747e08271dc497140af6b" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38691 } }
I20260925 02:48:41.592800 23215 raft_consensus.cc:374] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:41.592896 23215 raft_consensus.cc:729] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bacfe83465f747e08271dc497140af6b, State: Initialized, Role: FOLLOWER
I20260925 02:48:41.593015 23213 raft_consensus.cc:374] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:41.593147 23213 raft_consensus.cc:729] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6d942b1e58f47c1881cdd7a6dde18cc, State: Initialized, Role: FOLLOWER
I20260925 02:48:41.593623 23215 consensus_queue.cc:260] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6181514be64df7b197c533d7e56804" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 40801 } } peers { permanent_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35977 } } peers { permanent_uuid: "bacfe83465f747e08271dc497140af6b" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38691 } }
I20260925 02:48:41.593830 23213 consensus_queue.cc:260] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6181514be64df7b197c533d7e56804" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 40801 } } peers { permanent_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35977 } } peers { permanent_uuid: "bacfe83465f747e08271dc497140af6b" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38691 } }
I20260925 02:48:41.594826 23194 heartbeater.cc:503] Master 127.5.56.190:44499 was elected leader, sending a full tablet report...
I20260925 02:48:41.595583 23215 ts_tablet_manager.cc:1428] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: Time spent starting tablet: real 0.010s user 0.008s sys 0.003s
I20260925 02:48:41.596207 23214 raft_consensus.cc:348] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6181514be64df7b197c533d7e56804" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 40801 } } peers { permanent_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35977 } } peers { permanent_uuid: "bacfe83465f747e08271dc497140af6b" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38691 } }
I20260925 02:48:41.596728 23214 raft_consensus.cc:374] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:41.596884 23214 raft_consensus.cc:729] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb6181514be64df7b197c533d7e56804, State: Initialized, Role: FOLLOWER
I20260925 02:48:41.597539 23213 ts_tablet_manager.cc:1428] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: Time spent starting tablet: real 0.016s user 0.012s sys 0.002s
I20260925 02:48:41.597673 23214 consensus_queue.cc:260] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6181514be64df7b197c533d7e56804" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 40801 } } peers { permanent_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35977 } } peers { permanent_uuid: "bacfe83465f747e08271dc497140af6b" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38691 } }
I20260925 02:48:41.599013 23059 heartbeater.cc:503] Master 127.5.56.190:44499 was elected leader, sending a full tablet report...
I20260925 02:48:41.599628 23214 ts_tablet_manager.cc:1428] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
W20260925 02:48:41.632593 22923 tablet.cc:2369] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:48:41.666545 23060 tablet.cc:2369] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:41.676487 23220 raft_consensus.cc:482] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:41.676872 23220 raft_consensus.cc:504] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6181514be64df7b197c533d7e56804" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 40801 } } peers { permanent_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35977 } } peers { permanent_uuid: "bacfe83465f747e08271dc497140af6b" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38691 } }
I20260925 02:48:41.678684 23220 leader_election.cc:288] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb6181514be64df7b197c533d7e56804 (127.5.56.130:40801), bacfe83465f747e08271dc497140af6b (127.5.56.131:38691)
I20260925 02:48:41.685904 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.679362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:39739 (local address 127.5.56.131:38691)
0925 02:48:41.679628 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.679634 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.680531 (+ 897us) server_negotiation.cc:407] Connection header received
0925 02:48:41.680703 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.680707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.680791 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.680895 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:41.681512 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.682261 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.684130 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.684908 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.685073 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.685167 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:41.685495 (+ 328us) server_negotiation.cc:299] Negotiation successful
0925 02:48:41.685630 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:48:41.685938 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.679164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:57125 (local address 127.5.56.130:40801)
0925 02:48:41.679399 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.679407 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.680164 (+ 757us) server_negotiation.cc:407] Connection header received
0925 02:48:41.680405 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.680412 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.680504 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.680601 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:41.681338 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.682115 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.684157 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.684925 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.685105 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.685179 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:41.685551 (+ 372us) server_negotiation.cc:299] Negotiation successful
0925 02:48:41.685683 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:48:41.686023 23223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.679583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:38691 (local address 127.5.56.129:39739)
0925 02:48:41.680411 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:41.680427 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:41.680559 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:41.680920 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:41.680923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:41.680943 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:41.681348 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.681361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.682459 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.682466 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.683968 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.683980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.685135 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.685141 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.685315 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.685354 (+ 39us) client_negotiation.cc:769] Sending connection context
0925 02:48:41.685428 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 02:48:41.685571 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"mutex_wait_us":33,"thread_start_us":429,"threads_started":1}
I20260925 02:48:41.686075 23222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.679164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:40801 (local address 127.5.56.129:57125)
0925 02:48:41.680019 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:41.680046 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:41.680234 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:41.680652 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:41.680658 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:41.680695 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:41.681182 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.681194 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.682247 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.682252 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.683981 (+ 1729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:41.683993 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.685176 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.685185 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:41.685369 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.685394 (+ 25us) client_negotiation.cc:769] Sending connection context
0925 02:48:41.685507 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 02:48:41.685633 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":183,"thread_start_us":422,"threads_started":1}
I20260925 02:48:41.687302 23013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4701692bd7c4ba7ad0637baec4f5e41" candidate_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6181514be64df7b197c533d7e56804" is_pre_election: true
I20260925 02:48:41.687309 23149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4701692bd7c4ba7ad0637baec4f5e41" candidate_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bacfe83465f747e08271dc497140af6b" is_pre_election: true
I20260925 02:48:41.687804 23149 raft_consensus.cc:2454] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6d942b1e58f47c1881cdd7a6dde18cc in term 0.
I20260925 02:48:41.687805 23013 raft_consensus.cc:2454] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6d942b1e58f47c1881cdd7a6dde18cc in term 0.
I20260925 02:48:41.688491 22811 leader_election.cc:302] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: bacfe83465f747e08271dc497140af6b, bb6181514be64df7b197c533d7e56804, c6d942b1e58f47c1881cdd7a6dde18cc; no voters:
I20260925 02:48:41.688926 23220 raft_consensus.cc:2781] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:41.689077 23220 raft_consensus.cc:482] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:41.689199 23220 raft_consensus.cc:3037] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:41.691545 23220 raft_consensus.cc:504] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6181514be64df7b197c533d7e56804" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 40801 } } peers { permanent_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35977 } } peers { permanent_uuid: "bacfe83465f747e08271dc497140af6b" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38691 } }
I20260925 02:48:41.692239 23220 leader_election.cc:288] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [CANDIDATE]: Term 1 election: Requested vote from peers bb6181514be64df7b197c533d7e56804 (127.5.56.130:40801), bacfe83465f747e08271dc497140af6b (127.5.56.131:38691)
I20260925 02:48:41.692703 23013 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4701692bd7c4ba7ad0637baec4f5e41" candidate_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb6181514be64df7b197c533d7e56804"
I20260925 02:48:41.692752 23149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4701692bd7c4ba7ad0637baec4f5e41" candidate_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bacfe83465f747e08271dc497140af6b"
I20260925 02:48:41.692945 23013 raft_consensus.cc:3037] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:41.692970 23149 raft_consensus.cc:3037] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:41.695484 23149 raft_consensus.cc:2454] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6d942b1e58f47c1881cdd7a6dde18cc in term 1.
I20260925 02:48:41.695508 23013 raft_consensus.cc:2454] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6d942b1e58f47c1881cdd7a6dde18cc in term 1.
I20260925 02:48:41.695957 22810 leader_election.cc:302] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bacfe83465f747e08271dc497140af6b, c6d942b1e58f47c1881cdd7a6dde18cc; no voters:
I20260925 02:48:41.696342 23220 raft_consensus.cc:2781] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:41.696792 23220 raft_consensus.cc:686] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 1 LEADER]: Becoming Leader. State: Replica: c6d942b1e58f47c1881cdd7a6dde18cc, State: Running, Role: LEADER
I20260925 02:48:41.697288 23220 consensus_queue.cc:237] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6181514be64df7b197c533d7e56804" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 40801 } } peers { permanent_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35977 } } peers { permanent_uuid: "bacfe83465f747e08271dc497140af6b" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38691 } }
I20260925 02:48:41.702301 22731 catalog_manager.cc:5495] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc reported cstate change: term changed from 0 to 1, leader changed from <none> to c6d942b1e58f47c1881cdd7a6dde18cc (127.5.56.129). New cstate: current_term: 1 leader_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb6181514be64df7b197c533d7e56804" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 40801 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6d942b1e58f47c1881cdd7a6dde18cc" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 35977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bacfe83465f747e08271dc497140af6b" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38691 } health_report { overall_health: UNKNOWN } } }
W20260925 02:48:41.718287 23195 tablet.cc:2369] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:41.745689 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.740130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.5.56.129:35977)
0925 02:48:41.740381 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.740390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.740470 (+ 80us) server_negotiation.cc:407] Connection header received
0925 02:48:41.740718 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.740722 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.740825 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.740979 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:41.741352 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.742075 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.743518 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.743925 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.744056 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.744577 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:41.745107 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:41.745277 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:41.745329 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 02:48:41.745412 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260925 02:48:41.754594 23149 raft_consensus.cc:1261] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b [term 1 FOLLOWER]: Refusing update from remote peer c6d942b1e58f47c1881cdd7a6dde18cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:41.754710 23013 raft_consensus.cc:1261] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804 [term 1 FOLLOWER]: Refusing update from remote peer c6d942b1e58f47c1881cdd7a6dde18cc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:41.755759 23220 consensus_queue.cc:1035] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bacfe83465f747e08271dc497140af6b" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:41.756085 23224 consensus_queue.cc:1035] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb6181514be64df7b197c533d7e56804" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 40801 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:41.786713 22733 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:41.787022 22733 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42798:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:41.791852 22733 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 40312d6498f4495696de0eec5320f676: Sending DeleteTablet for 3 replicas of tablet a4701692bd7c4ba7ad0637baec4f5e41
I20260925 02:48:41.793200 22993 tablet_service.cc:1514] Processing DeleteTablet for tablet a4701692bd7c4ba7ad0637baec4f5e41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:41 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:38878
I20260925 02:48:41.793749 23129 tablet_service.cc:1514] Processing DeleteTablet for tablet a4701692bd7c4ba7ad0637baec4f5e41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:41 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:38216
I20260925 02:48:41.794308 23237 tablet_replica.cc:330] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: stopping tablet replica
I20260925 02:48:41.794780 23237 raft_consensus.cc:2229] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:41.794996 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 22794
I20260925 02:48:41.795321 23237 raft_consensus.cc:2258] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:41.793915 22857 tablet_service.cc:1514] Processing DeleteTablet for tablet a4701692bd7c4ba7ad0637baec4f5e41 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:41 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:35230
I20260925 02:48:41.803333 23237 ts_tablet_manager.cc:1905] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:41.810842 23237 ts_tablet_manager.cc:1918] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:41.811134 23237 log.cc:1196] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/a4701692bd7c4ba7ad0637baec4f5e41
I20260925 02:48:41.811767 23237 ts_tablet_manager.cc:1939] T a4701692bd7c4ba7ad0637baec4f5e41 P bb6181514be64df7b197c533d7e56804: Deleting consensus metadata
I20260925 02:48:41.815958 22718 catalog_manager.cc:4841] TS bb6181514be64df7b197c533d7e56804 (127.5.56.130:40801): tablet a4701692bd7c4ba7ad0637baec4f5e41 (table test-table [id=f7b23fcefbc344efa97fec61238af37d]) successfully deleted
I20260925 02:48:41.819621 23239 tablet_replica.cc:330] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: stopping tablet replica
I20260925 02:48:41.820142 23239 raft_consensus.cc:2229] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:41.820706 23239 raft_consensus.cc:2258] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:41.821606 23238 tablet_replica.cc:330] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: stopping tablet replica
I20260925 02:48:41.822125 23238 raft_consensus.cc:2229] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:41.822925 23238 raft_consensus.cc:2258] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:41.823904 23239 ts_tablet_manager.cc:1905] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:41.825188 23238 ts_tablet_manager.cc:1905] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:41.831666 23239 ts_tablet_manager.cc:1918] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:41.831909 23239 log.cc:1196] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/a4701692bd7c4ba7ad0637baec4f5e41
I20260925 02:48:41.832530 23239 ts_tablet_manager.cc:1939] T a4701692bd7c4ba7ad0637baec4f5e41 P bacfe83465f747e08271dc497140af6b: Deleting consensus metadata
Sep 25 02:48:41 dist-test-slave-n563 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:48:41.833292 23238 ts_tablet_manager.cc:1918] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:41.833510 23238 log.cc:1196] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/a4701692bd7c4ba7ad0637baec4f5e41
I20260925 02:48:41.834342 23238 ts_tablet_manager.cc:1939] T a4701692bd7c4ba7ad0637baec4f5e41 P c6d942b1e58f47c1881cdd7a6dde18cc: Deleting consensus metadata
I20260925 02:48:41.835032 22717 catalog_manager.cc:4841] TS bacfe83465f747e08271dc497140af6b (127.5.56.131:38691): tablet a4701692bd7c4ba7ad0637baec4f5e41 (table test-table [id=f7b23fcefbc344efa97fec61238af37d]) successfully deleted
I20260925 02:48:41.836413 22720 catalog_manager.cc:4841] TS c6d942b1e58f47c1881cdd7a6dde18cc (127.5.56.129:35977): tablet a4701692bd7c4ba7ad0637baec4f5e41 (table test-table [id=f7b23fcefbc344efa97fec61238af37d]) successfully deleted
I20260925 02:48:41.841698 23210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.797409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35238 (local address 127.5.56.129:35977)
0925 02:48:41.797672 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.797681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.819042 (+ 21361us) server_negotiation.cc:407] Connection header received
0925 02:48:41.821597 (+ 2555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.821603 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.821697 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.821822 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:41.823645 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.824826 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.827154 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.827906 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.828008 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.833939 (+ 5931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:41.833956 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:41.833975 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:41.834032 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:41.838569 (+ 4537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:41.839322 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:41.839331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:41.839335 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:41.839527 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:41.839933 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:41.839939 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:41.839944 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:41.840413 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:41.840724 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:41.841005 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 02:48:41.841284 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:48:41.842518 22917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:41.945917 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 22794
I20260925 02:48:41.962419 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 22928
Sep 25 02:48:41 dist-test-slave-n563 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:48:41.977874 23208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:41.963219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.5.56.130:40801)
0925 02:48:41.963399 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:41.963404 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:41.963422 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:48:41.963562 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:41.963567 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:41.963646 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:41.963729 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:41.965020 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.965750 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.967278 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:41.967660 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:41.967754 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:41.972050 (+ 4296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:41.972066 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:41.972082 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:41.972123 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:41.975003 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:41.975730 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:41.975747 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:41.975749 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:41.975880 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:41.976256 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:41.976262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:41.976265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:41.976658 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:41.976989 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:41.977297 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 02:48:41.977522 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:48:41.978629 23054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:42.069129 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 22928
I20260925 02:48:42.086854 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 23065
Sep 25 02:48:42 dist-test-slave-n563 krb5kdc[22688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304519, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:48:42.102108 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:42.087694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38220 (local address 127.5.56.131:38691)
0925 02:48:42.087934 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:42.087941 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:42.087962 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:48:42.088154 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:42.088159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:42.088237 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:42.088348 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:42.089721 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.090526 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.091872 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.092281 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.092425 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:42.096994 (+ 4569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:42.097012 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:42.097029 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:42.097063 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:42.099743 (+ 2680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:42.100406 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:42.100413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:42.100416 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:42.100548 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:42.100868 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:42.100871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:42.100873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:42.101204 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:42.101427 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:42.101660 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:48:42.101833 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 02:48:42.102797 23189 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:42.192796 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 23065
I20260925 02:48:42.210631 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 22701
I20260925 02:48:42.211826 22763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:42.324635 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 22701
2026-09-25T02:48:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4466 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T02:48:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:42Z Disabled control of system clock
I20260925 02:48:42.409581 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:33887
--webserver_interface=127.5.56.190
--webserver_port=0
--builtin_ntp_servers=127.5.56.148:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:33887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:48:42.753161 23248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:42.753542 23248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:42.753679 23248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:42.761066 23248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:42.761188 23248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:42.761226 23248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:42.761283 23248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:42.761328 23248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:42.770886 23248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:33887
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:33887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:42.772195 23248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:42.774091 23248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:42.786549 23253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:42.786626 23254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:42.786877 23248 server_base.cc:1034] running on GCE node
W20260925 02:48:42.786525 23256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:42.788039 23248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:42.789667 23248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:42.790885 23248 hybrid_clock.cc:648] HybridClock initialized: now 1790304522790842 us; error 30 us; skew 500 ppm
I20260925 02:48:42.791206 23248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:42.793408 23248 webserver.cc:460] Webserver started at http://127.5.56.190:35603/ using document root <none> and password file <none>
I20260925 02:48:42.794080 23248 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:42.794173 23248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:42.794581 23248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:42.797587 23248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "721f82f36f444664bf73cd3c367f16f6"
format_stamp: "Formatted at 2026-09-25 02:48:42 on dist-test-slave-n563"
server_key: "3539a8a7bf9facaae736c978da00b478"
server_key_iv: "f11f4baeb4ced71c9f26c88973083d69"
server_key_version: "encryptionkey@0"
I20260925 02:48:42.798369 23248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "721f82f36f444664bf73cd3c367f16f6"
format_stamp: "Formatted at 2026-09-25 02:48:42 on dist-test-slave-n563"
server_key: "3539a8a7bf9facaae736c978da00b478"
server_key_iv: "f11f4baeb4ced71c9f26c88973083d69"
server_key_version: "encryptionkey@0"
I20260925 02:48:42.803740 23248 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:48:42.807528 23262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:42.808956 23248 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:48:42.809149 23248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "721f82f36f444664bf73cd3c367f16f6"
format_stamp: "Formatted at 2026-09-25 02:48:42 on dist-test-slave-n563"
server_key: "3539a8a7bf9facaae736c978da00b478"
server_key_iv: "f11f4baeb4ced71c9f26c88973083d69"
server_key_version: "encryptionkey@0"
I20260925 02:48:42.809414 23248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:42.830683 23248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:42.831597 23248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:42.831918 23248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:42.851872 23248 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:33887
I20260925 02:48:42.851892 23314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:33887 every 8 connection(s)
I20260925 02:48:42.853353 23248 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:42.855868 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 23248
I20260925 02:48:42.856047 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:48:42.856626 5346 external_mini_cluster.cc:1428] Setting key 1f13828d95b58680cd1ce352f02a9e52
I20260925 02:48:42.858063 23315 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:42.870142 23318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:42.858674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58834 (local address 127.5.56.190:33887)
0925 02:48:42.859676 (+ 1002us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:42.859687 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:42.859728 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:48:42.861347 (+ 1619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:42.861370 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:42.861843 (+ 473us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:42.862280 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:42.862884 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.864214 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.865613 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:42.866288 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:42.866524 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:42.867764 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:42.867771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:42.867785 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:42.868380 (+ 595us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:42.868458 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:42.868486 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:42.868630 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:42.868855 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 02:48:42.869169 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":357,"threads_started":1}
I20260925 02:48:42.870641 23315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6: Bootstrap starting.
I20260925 02:48:42.874358 23315 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:42.875490 23315 log.cc:824] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:42.878635 23315 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6: No bootstrap required, opened a new log
I20260925 02:48:42.886536 23315 raft_consensus.cc:348] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721f82f36f444664bf73cd3c367f16f6" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33887 } }
I20260925 02:48:42.886977 23315 raft_consensus.cc:374] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:42.887094 23315 raft_consensus.cc:729] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 721f82f36f444664bf73cd3c367f16f6, State: Initialized, Role: FOLLOWER
I20260925 02:48:42.887686 23315 consensus_queue.cc:260] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721f82f36f444664bf73cd3c367f16f6" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33887 } }
I20260925 02:48:42.887935 23315 raft_consensus.cc:388] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:42.888085 23315 raft_consensus.cc:482] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:42.888269 23315 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:42.890619 23315 raft_consensus.cc:504] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721f82f36f444664bf73cd3c367f16f6" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33887 } }
I20260925 02:48:42.891135 23315 leader_election.cc:302] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 721f82f36f444664bf73cd3c367f16f6; no voters:
I20260925 02:48:42.891796 23315 leader_election.cc:288] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:42.891942 23320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:42.893208 23320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [term 1 LEADER]: Becoming Leader. State: Replica: 721f82f36f444664bf73cd3c367f16f6, State: Running, Role: LEADER
I20260925 02:48:42.893724 23320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721f82f36f444664bf73cd3c367f16f6" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33887 } }
I20260925 02:48:42.894004 23315 sys_catalog.cc:564] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:42.896751 23322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 721f82f36f444664bf73cd3c367f16f6. Latest consensus state: current_term: 1 leader_uuid: "721f82f36f444664bf73cd3c367f16f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721f82f36f444664bf73cd3c367f16f6" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33887 } } }
I20260925 02:48:42.897171 23322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:42.896847 23321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "721f82f36f444664bf73cd3c367f16f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "721f82f36f444664bf73cd3c367f16f6" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33887 } } }
I20260925 02:48:42.897604 23321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:42.905226 23327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:42.910627 23327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:42.919554 23327 catalog_manager.cc:1347] Generated new cluster ID: d92f0f2f49b34871bfe5599c306f4800
I20260925 02:48:42.919692 23327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:42.937438 23327 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:42.938911 23327 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:42.946697 23327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6: Generated new TSK 0
I20260925 02:48:42.947629 23327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:48:42.958693 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33887
--builtin_ntp_servers=127.5.56.148:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:48:43.328178 23339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:43.328480 23339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:43.328573 23339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:43.336222 23339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:43.336338 23339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:43.336448 23339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:48:43.345925 23339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:43.347275 23339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:43.349036 23339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:43.362464 23345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:43.362493 23344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:43.362703 23347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:43.363353 23339 server_base.cc:1034] running on GCE node
I20260925 02:48:43.364020 23339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:43.365166 23339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:43.366464 23339 hybrid_clock.cc:648] HybridClock initialized: now 1790304523366358 us; error 42 us; skew 500 ppm
I20260925 02:48:43.366791 23339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:43.369144 23339 webserver.cc:460] Webserver started at http://127.5.56.129:44901/ using document root <none> and password file <none>
I20260925 02:48:43.369903 23339 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:43.370041 23339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:43.370376 23339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:43.374734 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "436c035105484f98b5dc18ed5d7f3aed"
format_stamp: "Formatted at 2026-09-25 02:48:43 on dist-test-slave-n563"
server_key: "16ac926788cdd3fa0224f87c00ecf20e"
server_key_iv: "bd9a07fc79a166f875bf0781372cf9b7"
server_key_version: "encryptionkey@0"
I20260925 02:48:43.375777 23339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "436c035105484f98b5dc18ed5d7f3aed"
format_stamp: "Formatted at 2026-09-25 02:48:43 on dist-test-slave-n563"
server_key: "16ac926788cdd3fa0224f87c00ecf20e"
server_key_iv: "bd9a07fc79a166f875bf0781372cf9b7"
server_key_version: "encryptionkey@0"
I20260925 02:48:43.382817 23339 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260925 02:48:43.386647 23353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:43.388110 23339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:48:43.388253 23339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "436c035105484f98b5dc18ed5d7f3aed"
format_stamp: "Formatted at 2026-09-25 02:48:43 on dist-test-slave-n563"
server_key: "16ac926788cdd3fa0224f87c00ecf20e"
server_key_iv: "bd9a07fc79a166f875bf0781372cf9b7"
server_key_version: "encryptionkey@0"
I20260925 02:48:43.388476 23339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:43.414758 23339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:43.415608 23339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:43.415889 23339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:43.416847 23339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:43.418316 23339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:43.418457 23339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:43.418560 23339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:43.418620 23339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:43.452464 23339 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:41191
I20260925 02:48:43.452574 23466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:41191 every 8 connection(s)
I20260925 02:48:43.453863 23339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:43.463165 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 23339
I20260925 02:48:43.463342 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:43.463922 5346 external_mini_cluster.cc:1428] Setting key 3c86b84da2e7f9d0280ed2562ac6d824
I20260925 02:48:43.467216 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:43.456490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:33963 (local address 127.5.56.190:33887)
0925 02:48:43.456988 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:43.456993 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:43.458056 (+ 1063us) server_negotiation.cc:407] Connection header received
0925 02:48:43.459948 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:43.459953 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:43.460049 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:43.460169 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:43.461250 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.461972 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.464186 (+ 2214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.464602 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.464768 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:43.466424 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:43.466432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:43.466437 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:43.466461 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:43.466513 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:43.466527 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:43.466600 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:43.466871 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 02:48:43.466952 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":280,"threads_started":1}
I20260925 02:48:43.467262 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33887
--builtin_ntp_servers=127.5.56.148:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:48:43.468858 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:43.457002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:33887 (local address 127.5.56.129:33963)
0925 02:48:43.457891 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:43.457936 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:43.459668 (+ 1732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:43.460412 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:43.460420 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:43.460514 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:43.461075 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:43.461092 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.462147 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.462151 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:43.464033 (+ 1882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:43.464049 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.464847 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.464852 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:43.465151 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:43.465999 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:43.466002 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:43.466063 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:43.466181 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:43.466630 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:43.466643 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:43.466651 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 02:48:43.466869 (+ 218us) client_negotiation.cc:240] Negotiation successful
0925 02:48:43.467311 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":267,"threads_started":1}
I20260925 02:48:43.470573 23467 heartbeater.cc:347] Connected to a master server at 127.5.56.190:33887
I20260925 02:48:43.470913 23467 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:43.471679 23467 heartbeater.cc:511] Master 127.5.56.190:33887 requested a full tablet report, sending...
I20260925 02:48:43.474133 23280 ts_manager.cc:194] Registered new tserver with Master: 436c035105484f98b5dc18ed5d7f3aed (127.5.56.129:41191)
I20260925 02:48:43.476161 23280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.129:33963
W20260925 02:48:43.804857 23471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:43.805197 23471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:43.805359 23471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:43.813021 23471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:43.813182 23471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:43.813391 23471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:48:43.822896 23471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:43.824294 23471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:43.826085 23471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:43.839845 23476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:43.839887 23477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:43.839918 23479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:43.840597 23471 server_base.cc:1034] running on GCE node
I20260925 02:48:43.841249 23471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:43.842278 23471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:43.843513 23471 hybrid_clock.cc:648] HybridClock initialized: now 1790304523843472 us; error 75 us; skew 500 ppm
I20260925 02:48:43.843845 23471 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:43.846060 23471 webserver.cc:460] Webserver started at http://127.5.56.130:46333/ using document root <none> and password file <none>
I20260925 02:48:43.846836 23471 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:43.846967 23471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:43.847293 23471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:43.850240 23471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "a390cd437afb4510b197bbc739cdd552"
format_stamp: "Formatted at 2026-09-25 02:48:43 on dist-test-slave-n563"
server_key: "027ebe434fb79c4b7bc5b0acb264e836"
server_key_iv: "0850c5f2362749d52ec8a893aaa2ba42"
server_key_version: "encryptionkey@0"
I20260925 02:48:43.851130 23471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "a390cd437afb4510b197bbc739cdd552"
format_stamp: "Formatted at 2026-09-25 02:48:43 on dist-test-slave-n563"
server_key: "027ebe434fb79c4b7bc5b0acb264e836"
server_key_iv: "0850c5f2362749d52ec8a893aaa2ba42"
server_key_version: "encryptionkey@0"
I20260925 02:48:43.856460 23471 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:48:43.860121 23485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:43.861548 23471 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:48:43.861737 23471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "a390cd437afb4510b197bbc739cdd552"
format_stamp: "Formatted at 2026-09-25 02:48:43 on dist-test-slave-n563"
server_key: "027ebe434fb79c4b7bc5b0acb264e836"
server_key_iv: "0850c5f2362749d52ec8a893aaa2ba42"
server_key_version: "encryptionkey@0"
I20260925 02:48:43.861974 23471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:43.883258 23471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:43.884191 23471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:43.884526 23471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:43.885723 23471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:43.887372 23471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:43.887460 23471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:43.887547 23471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:43.887612 23471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:43.922156 23471 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:43523
I20260925 02:48:43.922250 23598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:43523 every 8 connection(s)
I20260925 02:48:43.923532 23471 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:43.926041 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 23471
I20260925 02:48:43.926208 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:43.926771 5346 external_mini_cluster.cc:1428] Setting key 28549469659db66151ef9a86984ec21c
I20260925 02:48:43.929620 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33887
--builtin_ntp_servers=127.5.56.148:32865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:48:43.936822 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:43.926285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:59959 (local address 127.5.56.190:33887)
0925 02:48:43.926547 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:43.926554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:43.928234 (+ 1680us) server_negotiation.cc:407] Connection header received
0925 02:48:43.930136 (+ 1902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:43.930141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:43.930236 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:43.930349 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:43.931513 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.932248 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.933848 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.934276 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.934435 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:43.935963 (+ 1528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:43.935969 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:43.935972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:43.935993 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:43.936032 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:43.936045 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:43.936146 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:43.936385 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 02:48:43.936493 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260925 02:48:43.937906 23601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:43.926939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:33887 (local address 127.5.56.130:59959)
0925 02:48:43.928062 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:43.928119 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:43.929841 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:43.930626 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:43.930635 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:43.930735 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:43.931332 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:43.931349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.932411 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.932416 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:43.933695 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:43.933708 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:43.934535 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:43.934543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:43.934791 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:43.935608 (+ 817us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:43.935610 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:43.935653 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:43.935744 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:43.936144 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:43.936157 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:43.936167 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 02:48:43.936360 (+ 193us) client_negotiation.cc:240] Negotiation successful
0925 02:48:43.936789 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":365,"threads_started":1}
I20260925 02:48:43.939625 23599 heartbeater.cc:347] Connected to a master server at 127.5.56.190:33887
I20260925 02:48:43.939965 23599 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:43.940817 23599 heartbeater.cc:511] Master 127.5.56.190:33887 requested a full tablet report, sending...
I20260925 02:48:43.942449 23280 ts_manager.cc:194] Registered new tserver with Master: a390cd437afb4510b197bbc739cdd552 (127.5.56.130:43523)
I20260925 02:48:43.943305 23280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.130:59959
W20260925 02:48:44.272801 23602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:44.273104 23602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:44.273274 23602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:44.280763 23602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:44.280885 23602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:44.281065 23602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:48:44.290625 23602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:32865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:33887
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:44.292011 23602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:44.294015 23602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:44.308274 23607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:44.308352 23610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:44.308316 23608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:44.309235 23602 server_base.cc:1034] running on GCE node
I20260925 02:48:44.309875 23602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:44.310969 23602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:44.312220 23602 hybrid_clock.cc:648] HybridClock initialized: now 1790304524312159 us; error 51 us; skew 500 ppm
I20260925 02:48:44.312578 23602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:44.315160 23602 webserver.cc:460] Webserver started at http://127.5.56.131:45473/ using document root <none> and password file <none>
I20260925 02:48:44.315929 23602 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:44.316063 23602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:44.316437 23602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:44.319479 23602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "5e5f9b14eb60416a8ce96001bd449172"
format_stamp: "Formatted at 2026-09-25 02:48:44 on dist-test-slave-n563"
server_key: "fd45637d4da3b1f8cb794f95902019a8"
server_key_iv: "a59176e594bdaccab3c41997cba862cf"
server_key_version: "encryptionkey@0"
I20260925 02:48:44.320287 23602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "5e5f9b14eb60416a8ce96001bd449172"
format_stamp: "Formatted at 2026-09-25 02:48:44 on dist-test-slave-n563"
server_key: "fd45637d4da3b1f8cb794f95902019a8"
server_key_iv: "a59176e594bdaccab3c41997cba862cf"
server_key_version: "encryptionkey@0"
I20260925 02:48:44.325656 23602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:48:44.329222 23616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:44.330662 23602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:44.330845 23602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "5e5f9b14eb60416a8ce96001bd449172"
format_stamp: "Formatted at 2026-09-25 02:48:44 on dist-test-slave-n563"
server_key: "fd45637d4da3b1f8cb794f95902019a8"
server_key_iv: "a59176e594bdaccab3c41997cba862cf"
server_key_version: "encryptionkey@0"
I20260925 02:48:44.331077 23602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:44.381057 23602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:44.382249 23602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:44.382674 23602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:44.384104 23602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:44.385627 23602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:44.385764 23602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:44.385890 23602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:44.385998 23602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:44.420042 23602 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:43665
I20260925 02:48:44.420132 23728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:43665 every 8 connection(s)
I20260925 02:48:44.421418 23602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:44.428215 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 23602
I20260925 02:48:44.428416 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:44.429015 5346 external_mini_cluster.cc:1428] Setting key d76f495767899bd2e15365bfba0a3382
I20260925 02:48:44.435128 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.424011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:52781 (local address 127.5.56.190:33887)
0925 02:48:44.424206 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.424210 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.425711 (+ 1501us) server_negotiation.cc:407] Connection header received
0925 02:48:44.427651 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.427656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.427759 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.427889 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:44.429608 (+ 1719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.430568 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.432493 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.432883 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.432980 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.434265 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:44.434270 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:44.434273 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:44.434293 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:44.434324 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:44.434336 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:44.434469 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:44.434703 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 02:48:44.434835 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:48:44.436276 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.424535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:33887 (local address 127.5.56.131:52781)
0925 02:48:44.425536 (+ 1001us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:44.425580 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:44.427354 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:44.428309 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:44.428323 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:44.428468 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:44.429386 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.429414 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.430730 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.430736 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.432358 (+ 1622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.432375 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.433068 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.433073 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.433270 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.433936 (+ 666us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:44.433938 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:44.433980 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:44.434083 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:44.434477 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:44.434485 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:44.434491 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 02:48:44.434687 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 02:48:44.435019 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":280,"threads_started":1}
I20260925 02:48:44.437608 23729 heartbeater.cc:347] Connected to a master server at 127.5.56.190:33887
I20260925 02:48:44.437889 23729 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:44.438663 23729 heartbeater.cc:511] Master 127.5.56.190:33887 requested a full tablet report, sending...
I20260925 02:48:44.440012 23280 ts_manager.cc:194] Registered new tserver with Master: 5e5f9b14eb60416a8ce96001bd449172 (127.5.56.131:43665)
I20260925 02:48:44.440824 23280 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.131:52781
I20260925 02:48:44.444427 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:48:44.454439 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.448524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58850 (local address 127.5.56.190:33887)
0925 02:48:44.448755 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.448764 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.449102 (+ 338us) server_negotiation.cc:407] Connection header received
0925 02:48:44.449250 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.449254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.449330 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.449422 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:44.449854 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.450600 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.451928 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.452529 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.452650 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.453683 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:44.453689 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:44.453694 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:44.453716 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:44.453755 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:44.453770 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:44.453855 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:44.453977 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 02:48:44.454071 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:48:44.458598 23280 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:44.465405 23280 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:44.479984 23467 heartbeater.cc:503] Master 127.5.56.190:33887 was elected leader, sending a full tablet report...
I20260925 02:48:44.490700 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.478087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:43523 (local address 127.0.0.1:38318)
0925 02:48:44.480767 (+ 2680us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:44.480802 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:44.480992 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:44.482032 (+ 1040us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:44.482042 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:44.482075 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:44.482592 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.482606 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.484992 (+ 2386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.484997 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.487288 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.487303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.489572 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.489627 (+ 55us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.489822 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.489882 (+ 60us) client_negotiation.cc:769] Sending connection context
0925 02:48:44.490035 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 02:48:44.490259 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2545,"mutex_wait_us":3,"thread_start_us":2376,"threads_started":1}
I20260925 02:48:44.490675 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.478141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.5.56.130:43523)
0925 02:48:44.479028 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.479035 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.480933 (+ 1898us) server_negotiation.cc:407] Connection header received
0925 02:48:44.481264 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.481275 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.481536 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.481707 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:44.483288 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.484840 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.487517 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.489011 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.489141 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.489448 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:44.490103 (+ 655us) server_negotiation.cc:299] Negotiation successful
0925 02:48:44.490280 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":720,"thread_start_us":615,"threads_started":1}
I20260925 02:48:44.491147 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.481020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:41191 (local address 127.0.0.1:42842)
0925 02:48:44.482744 (+ 1724us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:44.482760 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:44.482872 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:44.484018 (+ 1146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:44.484021 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:44.484041 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:44.484387 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.484398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.486496 (+ 2098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.486502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.488375 (+ 1873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.488392 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.490259 (+ 1867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.490266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.490540 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.490571 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 02:48:44.490657 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 02:48:44.490808 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1575,"mutex_wait_us":17,"thread_start_us":1401,"threads_started":1}
I20260925 02:48:44.492198 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.478164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.5.56.131:43665)
0925 02:48:44.479941 (+ 1777us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.479949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.481354 (+ 1405us) server_negotiation.cc:407] Connection header received
0925 02:48:44.481445 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.481451 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.481638 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.481819 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:44.484374 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.486040 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.488307 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.489694 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.489828 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.490106 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:44.491503 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0925 02:48:44.491660 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1612,"thread_start_us":1422,"threads_started":1}
I20260925 02:48:44.491937 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.477697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:43665 (local address 127.0.0.1:42634)
0925 02:48:44.480556 (+ 2859us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:44.481102 (+ 546us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:44.481215 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:44.482078 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:44.482082 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:44.482113 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:44.483033 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.483046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.486189 (+ 3143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.486194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.488117 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.488129 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.490259 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.490266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.491296 (+ 1030us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.491345 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 02:48:44.491440 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:48:44.491569 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2551,"mutex_wait_us":524,"spinlock_wait_cycles":5376,"thread_start_us":2402,"threads_started":1}
I20260925 02:48:44.491393 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.481780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.5.56.129:41191)
0925 02:48:44.483437 (+ 1657us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.483446 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.483471 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:48:44.483559 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.483566 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.483824 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.484010 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:44.484615 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.486325 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.488580 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.490031 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.490490 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.490786 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:44.490856 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 02:48:44.491012 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"thread_start_us":1318,"threads_started":1}
I20260925 02:48:44.495273 23533 tablet_service.cc:1467] Processing CreateTablet for tablet 41228263328045e68f5887649340ef57 (DEFAULT_TABLE table=test-table [id=1dd1e1f90d0f4da7a0d9487e560fac2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:44.495874 23401 tablet_service.cc:1467] Processing CreateTablet for tablet 41228263328045e68f5887649340ef57 (DEFAULT_TABLE table=test-table [id=1dd1e1f90d0f4da7a0d9487e560fac2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:44.495823 23664 tablet_service.cc:1467] Processing CreateTablet for tablet 41228263328045e68f5887649340ef57 (DEFAULT_TABLE table=test-table [id=1dd1e1f90d0f4da7a0d9487e560fac2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:44.496804 23533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41228263328045e68f5887649340ef57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:44.497112 23401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41228263328045e68f5887649340ef57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:44.497172 23664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41228263328045e68f5887649340ef57. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:44.511814 23749 tablet_bootstrap.cc:492] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: Bootstrap starting.
I20260925 02:48:44.514359 23748 tablet_bootstrap.cc:492] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: Bootstrap starting.
I20260925 02:48:44.516655 23749 tablet_bootstrap.cc:654] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:44.517791 23748 tablet_bootstrap.cc:654] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:44.517863 23749 log.cc:824] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:44.518805 23750 tablet_bootstrap.cc:492] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: Bootstrap starting.
I20260925 02:48:44.519037 23748 log.cc:824] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:44.521490 23750 tablet_bootstrap.cc:654] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:44.521763 23748 tablet_bootstrap.cc:492] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: No bootstrap required, opened a new log
I20260925 02:48:44.522065 23748 ts_tablet_manager.cc:1397] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260925 02:48:44.522710 23750 log.cc:824] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:44.524128 23749 tablet_bootstrap.cc:492] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: No bootstrap required, opened a new log
I20260925 02:48:44.524418 23749 ts_tablet_manager.cc:1397] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260925 02:48:44.525759 23750 tablet_bootstrap.cc:492] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: No bootstrap required, opened a new log
I20260925 02:48:44.526016 23750 ts_tablet_manager.cc:1397] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260925 02:48:44.530254 23748 raft_consensus.cc:348] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5f9b14eb60416a8ce96001bd449172" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43665 } } peers { permanent_uuid: "a390cd437afb4510b197bbc739cdd552" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43523 } } peers { permanent_uuid: "436c035105484f98b5dc18ed5d7f3aed" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41191 } }
I20260925 02:48:44.530715 23748 raft_consensus.cc:374] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:44.530822 23748 raft_consensus.cc:729] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 436c035105484f98b5dc18ed5d7f3aed, State: Initialized, Role: FOLLOWER
I20260925 02:48:44.531493 23748 consensus_queue.cc:260] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5f9b14eb60416a8ce96001bd449172" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43665 } } peers { permanent_uuid: "a390cd437afb4510b197bbc739cdd552" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43523 } } peers { permanent_uuid: "436c035105484f98b5dc18ed5d7f3aed" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41191 } }
I20260925 02:48:44.533318 23748 ts_tablet_manager.cc:1428] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260925 02:48:44.535386 23749 raft_consensus.cc:348] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5f9b14eb60416a8ce96001bd449172" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43665 } } peers { permanent_uuid: "a390cd437afb4510b197bbc739cdd552" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43523 } } peers { permanent_uuid: "436c035105484f98b5dc18ed5d7f3aed" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41191 } }
I20260925 02:48:44.535840 23749 raft_consensus.cc:374] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:44.535943 23749 raft_consensus.cc:729] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a390cd437afb4510b197bbc739cdd552, State: Initialized, Role: FOLLOWER
I20260925 02:48:44.536589 23749 consensus_queue.cc:260] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5f9b14eb60416a8ce96001bd449172" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43665 } } peers { permanent_uuid: "a390cd437afb4510b197bbc739cdd552" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43523 } } peers { permanent_uuid: "436c035105484f98b5dc18ed5d7f3aed" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41191 } }
I20260925 02:48:44.536870 23750 raft_consensus.cc:348] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5f9b14eb60416a8ce96001bd449172" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43665 } } peers { permanent_uuid: "a390cd437afb4510b197bbc739cdd552" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43523 } } peers { permanent_uuid: "436c035105484f98b5dc18ed5d7f3aed" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41191 } }
I20260925 02:48:44.537328 23750 raft_consensus.cc:374] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:44.537421 23750 raft_consensus.cc:729] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5f9b14eb60416a8ce96001bd449172, State: Initialized, Role: FOLLOWER
I20260925 02:48:44.537770 23599 heartbeater.cc:503] Master 127.5.56.190:33887 was elected leader, sending a full tablet report...
I20260925 02:48:44.537998 23750 consensus_queue.cc:260] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5f9b14eb60416a8ce96001bd449172" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43665 } } peers { permanent_uuid: "a390cd437afb4510b197bbc739cdd552" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43523 } } peers { permanent_uuid: "436c035105484f98b5dc18ed5d7f3aed" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41191 } }
I20260925 02:48:44.538558 23749 ts_tablet_manager.cc:1428] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
I20260925 02:48:44.539680 23729 heartbeater.cc:503] Master 127.5.56.190:33887 was elected leader, sending a full tablet report...
I20260925 02:48:44.540354 23750 ts_tablet_manager.cc:1428] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260925 02:48:44.649940 23756 raft_consensus.cc:482] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:44.650295 23756 raft_consensus.cc:504] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5f9b14eb60416a8ce96001bd449172" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43665 } } peers { permanent_uuid: "a390cd437afb4510b197bbc739cdd552" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43523 } } peers { permanent_uuid: "436c035105484f98b5dc18ed5d7f3aed" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41191 } }
I20260925 02:48:44.652132 23756 leader_election.cc:288] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a390cd437afb4510b197bbc739cdd552 (127.5.56.130:43523), 436c035105484f98b5dc18ed5d7f3aed (127.5.56.129:41191)
I20260925 02:48:44.658797 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.652716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:45737 (local address 127.5.56.130:43523)
0925 02:48:44.652922 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.652930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.652955 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:48:44.653197 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.653204 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.653311 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.653421 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:44.653934 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.654931 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.656811 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.657884 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.658010 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.658062 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:44.658499 (+ 437us) server_negotiation.cc:299] Negotiation successful
0925 02:48:44.658596 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:48:44.658931 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.652425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:43523 (local address 127.5.56.131:45737)
0925 02:48:44.652784 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:44.652809 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:44.652980 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:44.653472 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:44.653498 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:44.653533 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:44.653771 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.653783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.655067 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.655073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.656637 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.656650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.658043 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.658049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.658286 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.658331 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:48:44.658475 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 02:48:44.658623 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":2,"spinlock_wait_cycles":27008}
I20260925 02:48:44.659845 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.652630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:53815 (local address 127.5.56.129:41191)
0925 02:48:44.652852 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.652858 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.653534 (+ 676us) server_negotiation.cc:407] Connection header received
0925 02:48:44.653803 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.653810 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.653914 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.653991 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:44.654622 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.655530 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.657585 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.658779 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.658946 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.659026 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:44.659488 (+ 462us) server_negotiation.cc:299] Negotiation successful
0925 02:48:44.659615 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:48:44.659943 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.652425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:41191 (local address 127.5.56.131:53815)
0925 02:48:44.653409 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:44.653425 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:44.653624 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:44.654039 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:44.654044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:44.654068 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:44.654471 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.654483 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.655729 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.655733 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.657377 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:44.657391 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.659161 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.659167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:44.659335 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.659357 (+ 22us) client_negotiation.cc:769] Sending connection context
0925 02:48:44.659460 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:48:44.659580 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":41,"spinlock_wait_cycles":26368,"thread_start_us":543,"threads_started":1}
I20260925 02:48:44.660023 23554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41228263328045e68f5887649340ef57" candidate_uuid: "5e5f9b14eb60416a8ce96001bd449172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a390cd437afb4510b197bbc739cdd552" is_pre_election: true
I20260925 02:48:44.660651 23554 raft_consensus.cc:2454] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5f9b14eb60416a8ce96001bd449172 in term 0.
I20260925 02:48:44.661015 23422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41228263328045e68f5887649340ef57" candidate_uuid: "5e5f9b14eb60416a8ce96001bd449172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436c035105484f98b5dc18ed5d7f3aed" is_pre_election: true
I20260925 02:48:44.661481 23617 leader_election.cc:302] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5f9b14eb60416a8ce96001bd449172, a390cd437afb4510b197bbc739cdd552; no voters:
I20260925 02:48:44.661543 23422 raft_consensus.cc:2454] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5e5f9b14eb60416a8ce96001bd449172 in term 0.
I20260925 02:48:44.661917 23756 raft_consensus.cc:2781] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:44.662103 23756 raft_consensus.cc:482] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:44.662218 23756 raft_consensus.cc:3037] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:44.664451 23756 raft_consensus.cc:504] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5f9b14eb60416a8ce96001bd449172" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43665 } } peers { permanent_uuid: "a390cd437afb4510b197bbc739cdd552" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43523 } } peers { permanent_uuid: "436c035105484f98b5dc18ed5d7f3aed" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41191 } }
I20260925 02:48:44.665030 23756 leader_election.cc:288] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [CANDIDATE]: Term 1 election: Requested vote from peers a390cd437afb4510b197bbc739cdd552 (127.5.56.130:43523), 436c035105484f98b5dc18ed5d7f3aed (127.5.56.129:41191)
I20260925 02:48:44.665570 23554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41228263328045e68f5887649340ef57" candidate_uuid: "5e5f9b14eb60416a8ce96001bd449172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a390cd437afb4510b197bbc739cdd552"
I20260925 02:48:44.665625 23422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41228263328045e68f5887649340ef57" candidate_uuid: "5e5f9b14eb60416a8ce96001bd449172" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "436c035105484f98b5dc18ed5d7f3aed"
I20260925 02:48:44.665768 23554 raft_consensus.cc:3037] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:44.665942 23422 raft_consensus.cc:3037] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:44.668218 23554 raft_consensus.cc:2454] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5f9b14eb60416a8ce96001bd449172 in term 1.
I20260925 02:48:44.668283 23422 raft_consensus.cc:2454] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e5f9b14eb60416a8ce96001bd449172 in term 1.
I20260925 02:48:44.668673 23617 leader_election.cc:302] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e5f9b14eb60416a8ce96001bd449172, a390cd437afb4510b197bbc739cdd552; no voters:
I20260925 02:48:44.668965 23756 raft_consensus.cc:2781] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:44.669445 23756 raft_consensus.cc:686] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 1 LEADER]: Becoming Leader. State: Replica: 5e5f9b14eb60416a8ce96001bd449172, State: Running, Role: LEADER
I20260925 02:48:44.670018 23756 consensus_queue.cc:237] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5f9b14eb60416a8ce96001bd449172" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43665 } } peers { permanent_uuid: "a390cd437afb4510b197bbc739cdd552" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43523 } } peers { permanent_uuid: "436c035105484f98b5dc18ed5d7f3aed" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41191 } }
W20260925 02:48:44.673163 23730 tablet.cc:2369] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:44.675124 23280 catalog_manager.cc:5495] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5e5f9b14eb60416a8ce96001bd449172 (127.5.56.131). New cstate: current_term: 1 leader_uuid: "5e5f9b14eb60416a8ce96001bd449172" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e5f9b14eb60416a8ce96001bd449172" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 43665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a390cd437afb4510b197bbc739cdd552" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "436c035105484f98b5dc18ed5d7f3aed" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41191 } health_report { overall_health: UNKNOWN } } }
W20260925 02:48:44.675670 23600 tablet.cc:2369] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:48:44.706640 23468 tablet.cc:2369] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:44.727485 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.722070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.5.56.131:43665)
0925 02:48:44.722271 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.722276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.722299 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:48:44.722453 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.722457 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.722538 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.722692 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:44.723053 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.723784 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.725258 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.725659 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.725789 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.726334 (+ 545us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:44.726918 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:44.727077 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:44.727139 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 02:48:44.727220 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:48:44.735713 23422 raft_consensus.cc:1261] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed [term 1 FOLLOWER]: Refusing update from remote peer 5e5f9b14eb60416a8ce96001bd449172: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:44.735734 23554 raft_consensus.cc:1261] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552 [term 1 FOLLOWER]: Refusing update from remote peer 5e5f9b14eb60416a8ce96001bd449172: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:44.736833 23758 consensus_queue.cc:1035] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [LEADER]: Connected to new peer: Peer: permanent_uuid: "436c035105484f98b5dc18ed5d7f3aed" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:44.737222 23756 consensus_queue.cc:1035] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a390cd437afb4510b197bbc739cdd552" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 43523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:44.765763 23279 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:44.766086 23279 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:44.770195 23279 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 721f82f36f444664bf73cd3c367f16f6: Sending DeleteTablet for 3 replicas of tablet 41228263328045e68f5887649340ef57
I20260925 02:48:44.771484 23664 tablet_service.cc:1514] Processing DeleteTablet for tablet 41228263328045e68f5887649340ef57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:44 UTC) from {username='slave'} at 127.0.0.1:42634
I20260925 02:48:44.771507 23401 tablet_service.cc:1514] Processing DeleteTablet for tablet 41228263328045e68f5887649340ef57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:44 UTC) from {username='slave'} at 127.0.0.1:42842
I20260925 02:48:44.771507 23533 tablet_service.cc:1514] Processing DeleteTablet for tablet 41228263328045e68f5887649340ef57 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:44 UTC) from {username='slave'} at 127.0.0.1:38318
I20260925 02:48:44.772578 23774 tablet_replica.cc:330] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: stopping tablet replica
I20260925 02:48:44.773247 23773 tablet_replica.cc:330] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: stopping tablet replica
I20260925 02:48:44.773484 23772 tablet_replica.cc:330] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: stopping tablet replica
I20260925 02:48:44.773954 23772 raft_consensus.cc:2229] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:44.774191 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 23339
I20260925 02:48:44.774436 23773 raft_consensus.cc:2229] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:44.774727 23772 raft_consensus.cc:2258] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:44.773969 23774 raft_consensus.cc:2229] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:44.775367 23774 raft_consensus.cc:2258] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:44.775696 23773 raft_consensus.cc:2258] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:44.777099 23772 ts_tablet_manager.cc:1905] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:44.779011 23773 ts_tablet_manager.cc:1905] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:44.780104 23774 ts_tablet_manager.cc:1905] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:44.786952 23772 ts_tablet_manager.cc:1918] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:44.787173 23772 log.cc:1196] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/41228263328045e68f5887649340ef57
I20260925 02:48:44.787796 23772 ts_tablet_manager.cc:1939] T 41228263328045e68f5887649340ef57 P 5e5f9b14eb60416a8ce96001bd449172: Deleting consensus metadata
I20260925 02:48:44.789887 23266 catalog_manager.cc:4841] TS 5e5f9b14eb60416a8ce96001bd449172 (127.5.56.131:43665): tablet 41228263328045e68f5887649340ef57 (table test-table [id=1dd1e1f90d0f4da7a0d9487e560fac2c]) successfully deleted
I20260925 02:48:44.792704 23774 ts_tablet_manager.cc:1918] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:44.792901 23774 log.cc:1196] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/41228263328045e68f5887649340ef57
I20260925 02:48:44.793495 23774 ts_tablet_manager.cc:1939] T 41228263328045e68f5887649340ef57 P 436c035105484f98b5dc18ed5d7f3aed: Deleting consensus metadata
I20260925 02:48:44.793807 23773 ts_tablet_manager.cc:1918] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:44.793967 23773 log.cc:1196] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/41228263328045e68f5887649340ef57
I20260925 02:48:44.794749 23773 ts_tablet_manager.cc:1939] T 41228263328045e68f5887649340ef57 P a390cd437afb4510b197bbc739cdd552: Deleting consensus metadata
I20260925 02:48:44.795665 23266 catalog_manager.cc:4841] TS 436c035105484f98b5dc18ed5d7f3aed (127.5.56.129:41191): tablet 41228263328045e68f5887649340ef57 (table test-table [id=1dd1e1f90d0f4da7a0d9487e560fac2c]) successfully deleted
I20260925 02:48:44.795748 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.776855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42848 (local address 127.5.56.129:41191)
0925 02:48:44.777033 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.777039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.785800 (+ 8761us) server_negotiation.cc:407] Connection header received
0925 02:48:44.786317 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.786324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.786474 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.786564 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:44.787318 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.788473 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.791145 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.791782 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.791898 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.793685 (+ 1787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:44.793699 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:44.793722 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:44.794654 (+ 932us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:44.794725 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:44.794755 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:44.794870 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:44.795264 (+ 394us) server_negotiation.cc:299] Negotiation successful
0925 02:48:44.795393 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:48:44.796624 23263 catalog_manager.cc:4841] TS a390cd437afb4510b197bbc739cdd552 (127.5.56.130:43523): tablet 41228263328045e68f5887649340ef57 (table test-table [id=1dd1e1f90d0f4da7a0d9487e560fac2c]) successfully deleted
I20260925 02:48:44.797065 23462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:44.889065 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 23339
I20260925 02:48:44.909297 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 23471
I20260925 02:48:44.916815 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:44.910219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38334 (local address 127.5.56.130:43523)
0925 02:48:44.910456 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:44.910463 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:44.910485 (+ 22us) server_negotiation.cc:407] Connection header received
0925 02:48:44.910582 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:44.910587 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:44.910708 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:44.910834 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:44.911245 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.912033 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.913400 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:44.913823 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:44.913961 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:44.915295 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:44.915305 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:44.915320 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:44.916013 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:44.916064 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:44.916084 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:44.916214 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:44.916351 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 02:48:44.916469 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:48:44.917587 23594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:45.007570 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 23471
I20260925 02:48:45.033150 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 23602
I20260925 02:48:45.041291 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:45.034171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.5.56.131:43665)
0925 02:48:45.034354 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:45.034360 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:45.034398 (+ 38us) server_negotiation.cc:407] Connection header received
0925 02:48:45.034659 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:45.034665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:45.034838 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:45.034965 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:45.035449 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.036367 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.037830 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:45.038312 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:45.038562 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:45.039735 (+ 1173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:45.039759 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:45.039775 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:45.040443 (+ 668us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:45.040496 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:45.040518 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:45.040688 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:45.040819 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 02:48:45.040949 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:48:45.042239 23724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:45.166092 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 23602
I20260925 02:48:45.187367 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 23248
I20260925 02:48:45.188516 23310 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:45.304706 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 23248
2026-09-25T02:48:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2974 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:45 dist-test-slave-n563 krb5kdc[23782](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:45 dist-test-slave-n563 krb5kdc[23782](info): set up 2 sockets
Sep 25 02:48:45 dist-test-slave-n563 krb5kdc[23782](info): commencing operation
krb5kdc: starting...
W20260925 02:48:46.431643 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.084s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:48:46 dist-test-slave-n563 krb5kdc[23782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:48:46.702801 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:36407
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:36407
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:47.051751 23795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:47.052155 23795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:47.052320 23795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:47.059876 23795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:47.060040 23795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:47.060161 23795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:47.060276 23795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:47.060356 23795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:47.070745 23795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:36407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:36407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:47.072196 23795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:47.073562 23795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:47.085875 23803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:47.085959 23800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:47.085963 23801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:47.087131 23795 server_base.cc:1034] running on GCE node
I20260925 02:48:47.088094 23795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:47.089679 23795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:47.090941 23795 hybrid_clock.cc:648] HybridClock initialized: now 1790304527090882 us; error 51 us; skew 500 ppm
I20260925 02:48:47.091331 23795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:47 dist-test-slave-n563 krb5kdc[23782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304527, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:47.100699 23795 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:48:47.102516 23795 webserver.cc:460] Webserver started at http://127.5.56.190:44853/ using document root <none> and password file <none>
I20260925 02:48:47.103278 23795 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:47.103432 23795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:47.103766 23795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:47.107009 23795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "531831f813d94e5eb89298a9abb664ef"
format_stamp: "Formatted at 2026-09-25 02:48:47 on dist-test-slave-n563"
server_key: "c8f897763b28216f78a0ecbcb8c4ff47"
server_key_iv: "ffb143f87673b1286e41646f31c56a17"
server_key_version: "encryptionkey@0"
I20260925 02:48:47.107829 23795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "531831f813d94e5eb89298a9abb664ef"
format_stamp: "Formatted at 2026-09-25 02:48:47 on dist-test-slave-n563"
server_key: "c8f897763b28216f78a0ecbcb8c4ff47"
server_key_iv: "ffb143f87673b1286e41646f31c56a17"
server_key_version: "encryptionkey@0"
I20260925 02:48:47.113476 23795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:48:47.117161 23810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:47.118755 23795 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:48:47.118949 23795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "531831f813d94e5eb89298a9abb664ef"
format_stamp: "Formatted at 2026-09-25 02:48:47 on dist-test-slave-n563"
server_key: "c8f897763b28216f78a0ecbcb8c4ff47"
server_key_iv: "ffb143f87673b1286e41646f31c56a17"
server_key_version: "encryptionkey@0"
I20260925 02:48:47.119171 23795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:47.139640 23795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:47.144591 23795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:47.144924 23795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:47.164407 23795 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:36407
I20260925 02:48:47.164445 23862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:36407 every 8 connection(s)
I20260925 02:48:47.165918 23795 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:47.169327 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 23795
I20260925 02:48:47.169525 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:48:47.170002 23863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:47.170009 5346 external_mini_cluster.cc:1428] Setting key e2d2bd5c11020b45528ac69692eed56d
I20260925 02:48:47.185684 23863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef: Bootstrap starting.
Sep 25 02:48:47 dist-test-slave-n563 krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:47.189096 23863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:47.190256 23863 log.cc:824] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:47.193257 23863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef: No bootstrap required, opened a new log
I20260925 02:48:47.195103 23866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:47.171951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45602 (local address 127.5.56.190:36407)
0925 02:48:47.172767 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:47.172778 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:47.172824 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:48:47.174929 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:47.174954 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:47.175374 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:47.175733 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:47.177731 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:47.179175 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:47.181248 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:47.182019 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:47.182188 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:47.187412 (+ 5224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:47.187421 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:47.187437 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:47.187495 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:47.190792 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:47.191586 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:47.191593 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:47.191600 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:47.191779 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:47.192218 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:47.192224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:47.192229 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:47.192784 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:47.193092 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:47.193543 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 02:48:47.194033 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":365,"threads_started":1}
I20260925 02:48:47.201480 23863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531831f813d94e5eb89298a9abb664ef" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36407 } }
I20260925 02:48:47.201906 23863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:47.202019 23863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 531831f813d94e5eb89298a9abb664ef, State: Initialized, Role: FOLLOWER
I20260925 02:48:47.202731 23863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531831f813d94e5eb89298a9abb664ef" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36407 } }
I20260925 02:48:47.203063 23863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:47.203224 23863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:47.203413 23863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:47.205711 23863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531831f813d94e5eb89298a9abb664ef" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36407 } }
I20260925 02:48:47.206259 23863 leader_election.cc:302] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 531831f813d94e5eb89298a9abb664ef; no voters:
I20260925 02:48:47.206967 23863 leader_election.cc:288] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:47.207132 23868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:47.207986 23868 raft_consensus.cc:686] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [term 1 LEADER]: Becoming Leader. State: Replica: 531831f813d94e5eb89298a9abb664ef, State: Running, Role: LEADER
I20260925 02:48:47.208515 23868 consensus_queue.cc:237] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531831f813d94e5eb89298a9abb664ef" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36407 } }
I20260925 02:48:47.209379 23863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:47.212201 23870 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 531831f813d94e5eb89298a9abb664ef. Latest consensus state: current_term: 1 leader_uuid: "531831f813d94e5eb89298a9abb664ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531831f813d94e5eb89298a9abb664ef" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36407 } } }
I20260925 02:48:47.212272 23869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "531831f813d94e5eb89298a9abb664ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "531831f813d94e5eb89298a9abb664ef" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 36407 } } }
I20260925 02:48:47.212821 23870 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:47.212821 23869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:47.213846 23876 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:47.219681 23876 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:47.234447 23876 catalog_manager.cc:1347] Generated new cluster ID: 2c8a9245d5d74995a13747272fd98b4e
I20260925 02:48:47.234714 23876 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:47.252525 23876 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:47.254522 23876 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:47.267782 23876 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef: Generated new TSK 0
I20260925 02:48:47.269116 23876 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:48:47.338486 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36407
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:47.688520 23889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:47.688886 23889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:47.689050 23889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:47.696632 23889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:47.696816 23889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:47.696995 23889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:48:47.707226 23889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:47.708649 23889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:47.710168 23889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:47.724681 23894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:47.724655 23895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:47.724910 23897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:47.725509 23889 server_base.cc:1034] running on GCE node
I20260925 02:48:47.726166 23889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:47.727284 23889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:47.728523 23889 hybrid_clock.cc:648] HybridClock initialized: now 1790304527728472 us; error 45 us; skew 500 ppm
I20260925 02:48:47.728884 23889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:47 dist-test-slave-n563 krb5kdc[23782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304527, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:47.732167 23889 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:48:47.733790 23889 webserver.cc:460] Webserver started at http://127.5.56.129:42915/ using document root <none> and password file <none>
I20260925 02:48:47.734607 23889 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:47.734766 23889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:47.735106 23889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:47.738209 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "cb553106db7948abb85cbf5451fd900c"
format_stamp: "Formatted at 2026-09-25 02:48:47 on dist-test-slave-n563"
server_key: "a02cd84122c67ea99592d2c3bee79c78"
server_key_iv: "ad1b4f4035c475b16b9d768d1683cf88"
server_key_version: "encryptionkey@0"
I20260925 02:48:47.739145 23889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "cb553106db7948abb85cbf5451fd900c"
format_stamp: "Formatted at 2026-09-25 02:48:47 on dist-test-slave-n563"
server_key: "a02cd84122c67ea99592d2c3bee79c78"
server_key_iv: "ad1b4f4035c475b16b9d768d1683cf88"
server_key_version: "encryptionkey@0"
I20260925 02:48:47.744822 23889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:48:47.748533 23904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:47.749986 23889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:47.750178 23889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "cb553106db7948abb85cbf5451fd900c"
format_stamp: "Formatted at 2026-09-25 02:48:47 on dist-test-slave-n563"
server_key: "a02cd84122c67ea99592d2c3bee79c78"
server_key_iv: "ad1b4f4035c475b16b9d768d1683cf88"
server_key_version: "encryptionkey@0"
I20260925 02:48:47.750430 23889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:47.772629 23889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:47.776860 23889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:47.777184 23889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:47.778205 23889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:47.779662 23889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:47.779781 23889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:47.779944 23889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:47.780018 23889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:47.815055 23889 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:44667
I20260925 02:48:47.815096 24017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:44667 every 8 connection(s)
I20260925 02:48:47.816411 23889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:47.818847 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 23889
I20260925 02:48:47.819073 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:47.819532 5346 external_mini_cluster.cc:1428] Setting key 8a06f26b08ec5483bfb8f8e994cdb652
Sep 25 02:48:47 dist-test-slave-n563 krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304527, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:47.838773 24020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:47.819042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:55775 (local address 127.5.56.190:36407)
0925 02:48:47.822781 (+ 3739us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:47.822787 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:47.822804 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:48:47.822854 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:47.822858 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:47.822946 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:47.823043 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:47.825064 (+ 2021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:47.826269 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:47.827922 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:47.828620 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:47.828775 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:47.832592 (+ 3817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:47.832601 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:47.832609 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:47.832657 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:47.835317 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:47.836306 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:47.836309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:47.836311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:47.836428 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:47.836926 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:47.836929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:47.836931 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:47.837201 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:47.837313 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:47.838264 (+ 951us) server_negotiation.cc:299] Negotiation successful
0925 02:48:47.838504 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3611,"thread_start_us":3506,"threads_started":1}
I20260925 02:48:47.840370 24021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:47.819751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36407 (local address 127.5.56.129:55775)
0925 02:48:47.820669 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:47.820715 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:47.822524 (+ 1809us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:47.823365 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:47.823377 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:47.824204 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:47.824867 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:47.824891 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:47.826490 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:47.826494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:47.827763 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:47.827776 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:47.828837 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:47.828842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:47.829049 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:47.829778 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:47.829802 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:47.832329 (+ 2527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:47.835470 (+ 3141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:47.835483 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:47.835507 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:47.836182 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:47.836537 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:47.836545 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:47.836551 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:47.836804 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:47.837340 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:47.837352 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:47.837849 (+ 497us) client_negotiation.cc:769] Sending connection context
0925 02:48:47.838161 (+ 312us) client_negotiation.cc:240] Negotiation successful
0925 02:48:47.838824 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":383,"threads_started":1}
I20260925 02:48:47.842140 24018 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36407
I20260925 02:48:47.842514 24018 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:47.843437 24018 heartbeater.cc:511] Master 127.5.56.190:36407 requested a full tablet report, sending...
I20260925 02:48:47.845829 23827 ts_manager.cc:194] Registered new tserver with Master: cb553106db7948abb85cbf5451fd900c (127.5.56.129:44667)
I20260925 02:48:47.847385 23827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:55775
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:48:47.885797 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36407
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:48.228941 24024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:48.229245 24024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:48.229429 24024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:48.236864 24024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:48.236994 24024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:48.237226 24024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:48:48.247496 24024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:48.248952 24024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:48.250447 24024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:48.264393 24029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:48.264369 24032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:48.264319 24030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:48.265177 24024 server_base.cc:1034] running on GCE node
I20260925 02:48:48.265808 24024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:48.266847 24024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:48.268086 24024 hybrid_clock.cc:648] HybridClock initialized: now 1790304528268040 us; error 25 us; skew 500 ppm
I20260925 02:48:48.268599 24024 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:48 dist-test-slave-n563 krb5kdc[23782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304528, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:48.272210 24024 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:48:48.273803 24024 webserver.cc:460] Webserver started at http://127.5.56.130:38013/ using document root <none> and password file <none>
I20260925 02:48:48.274570 24024 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:48.274685 24024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:48.274972 24024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:48.277984 24024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "af1a0b298e9141bbad0fc65f56a463fa"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-n563"
server_key: "a25259aabc2e02c98910f22a8057b1fe"
server_key_iv: "a47e2d6dfb56aaaa22db927ecff874e9"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.278844 24024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "af1a0b298e9141bbad0fc65f56a463fa"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-n563"
server_key: "a25259aabc2e02c98910f22a8057b1fe"
server_key_iv: "a47e2d6dfb56aaaa22db927ecff874e9"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.284236 24024 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:48:48.287825 24039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.289287 24024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:48:48.289450 24024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "af1a0b298e9141bbad0fc65f56a463fa"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-n563"
server_key: "a25259aabc2e02c98910f22a8057b1fe"
server_key_iv: "a47e2d6dfb56aaaa22db927ecff874e9"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.289628 24024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:48.315253 24024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:48.320168 24024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:48.320429 24024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:48.321460 24024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:48.322963 24024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:48.323038 24024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.323133 24024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:48.323186 24024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.357354 24024 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:35971
I20260925 02:48:48.357402 24152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:35971 every 8 connection(s)
I20260925 02:48:48.358829 24024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:48.363219 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 24024
I20260925 02:48:48.363444 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:48.363926 5346 external_mini_cluster.cc:1428] Setting key 88787380960428e3a33ad800aa7d9bd4
Sep 25 02:48:48 dist-test-slave-n563 krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304528, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:48.379637 24155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.361523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:33905 (local address 127.5.56.190:36407)
0925 02:48:48.362039 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:48.362044 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:48.363031 (+ 987us) server_negotiation.cc:407] Connection header received
0925 02:48:48.365065 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:48.365070 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:48.365162 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:48.365295 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:48.367026 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.367777 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.369373 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.369811 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.369970 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.373569 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:48.373574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:48.373577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:48.373633 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:48.376345 (+ 2712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.377118 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.377128 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.377130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.377266 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.377719 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.377731 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.377733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.378006 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:48.378179 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:48.378994 (+ 815us) server_negotiation.cc:299] Negotiation successful
0925 02:48:48.379249 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":248,"threads_started":1}
I20260925 02:48:48.380703 24156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.362115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36407 (local address 127.5.56.130:33905)
0925 02:48:48.362864 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:48.362912 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:48.364769 (+ 1857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:48.365531 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:48.365542 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:48.366166 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:48.366851 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.366870 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.367935 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.367948 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.369197 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.369214 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.370079 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.370087 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.370323 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.371097 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:48.371122 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:48.373310 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:48.376514 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:48.376523 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:48.376541 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:48.376982 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:48.377384 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:48.377389 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:48.377393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:48.377588 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:48.378150 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:48.378158 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:48.378582 (+ 424us) client_negotiation.cc:769] Sending connection context
0925 02:48:48.378892 (+ 310us) client_negotiation.cc:240] Negotiation successful
0925 02:48:48.379457 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":323,"threads_started":1}
I20260925 02:48:48.382268 24153 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36407
I20260925 02:48:48.382663 24153 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:48.383574 24153 heartbeater.cc:511] Master 127.5.56.190:36407 requested a full tablet report, sending...
I20260925 02:48:48.385447 23827 ts_manager.cc:194] Registered new tserver with Master: af1a0b298e9141bbad0fc65f56a463fa (127.5.56.130:35971)
I20260925 02:48:48.386323 23827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:33905
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:48:48.425496 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36407
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:41525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:48.765782 24159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:48.766101 24159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:48.766202 24159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:48.773936 24159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:48.774065 24159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:48.774183 24159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:48:48.784443 24159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:41525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:36407
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:48.785804 24159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:48.787156 24159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:48.801522 24164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:48.801492 24167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:48.801600 24165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:48.807904 24159 server_base.cc:1034] running on GCE node
I20260925 02:48:48.808667 24159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:48.809845 24159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:48.811081 24159 hybrid_clock.cc:648] HybridClock initialized: now 1790304528811037 us; error 42 us; skew 500 ppm
I20260925 02:48:48.811440 24159 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:48 dist-test-slave-n563 krb5kdc[23782](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304528, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:48.814879 24159 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:48:48.818817 24159 webserver.cc:460] Webserver started at http://127.5.56.131:38653/ using document root <none> and password file <none>
I20260925 02:48:48.819585 24159 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:48.819742 24159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:48.820115 24159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:48.823268 24159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "d7432997b92a4c6986c67d92b3e31296"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-n563"
server_key: "fa311fe97bb2e295628e53b72967fdff"
server_key_iv: "0ff7af57097a66fca21d60c47f488d80"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.824129 24159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "d7432997b92a4c6986c67d92b3e31296"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-n563"
server_key: "fa311fe97bb2e295628e53b72967fdff"
server_key_iv: "0ff7af57097a66fca21d60c47f488d80"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.829450 24159 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:48:48.833112 24174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.834571 24159 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:48.834749 24159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "d7432997b92a4c6986c67d92b3e31296"
format_stamp: "Formatted at 2026-09-25 02:48:48 on dist-test-slave-n563"
server_key: "fa311fe97bb2e295628e53b72967fdff"
server_key_iv: "0ff7af57097a66fca21d60c47f488d80"
server_key_version: "encryptionkey@0"
I20260925 02:48:48.834971 24159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:48.851588 24018 heartbeater.cc:503] Master 127.5.56.190:36407 was elected leader, sending a full tablet report...
I20260925 02:48:48.855108 24159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:48.859797 24159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:48.860109 24159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:48.861207 24159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:48.862669 24159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:48.862772 24159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.862928 24159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:48.863003 24159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:48.897495 24159 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:36099
I20260925 02:48:48.897531 24287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:36099 every 8 connection(s)
I20260925 02:48:48.898967 24159 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:48.901746 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 24159
I20260925 02:48:48.901968 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:48.902480 5346 external_mini_cluster.cc:1428] Setting key d01b35c35198c8bf48a4799d034dd7d5
Sep 25 02:48:48 dist-test-slave-n563 krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304528, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:48.920188 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.901534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:52901 (local address 127.5.56.190:36407)
0925 02:48:48.902041 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:48.902047 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:48.903176 (+ 1129us) server_negotiation.cc:407] Connection header received
0925 02:48:48.905083 (+ 1907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:48.905090 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:48.905179 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:48.905294 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:48.907037 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.907800 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.909380 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.909855 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.909991 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.913486 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:48.913491 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:48.913494 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:48.913528 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:48.916794 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.917786 (+ 992us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.917791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.917794 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.917963 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.918441 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.918448 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.918455 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.918741 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:48.918874 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:48.919722 (+ 848us) server_negotiation.cc:299] Negotiation successful
0925 02:48:48.919914 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":223,"threads_started":1}
I20260925 02:48:48.921245 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.902114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:36407 (local address 127.5.56.131:52901)
0925 02:48:48.902995 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:48.903044 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:48.904784 (+ 1740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:48.905561 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:48.905570 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:48.906123 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:48.906859 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.906879 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.907955 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.907960 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.909223 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.909235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.910087 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.910094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.910333 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.911089 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:48.911124 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:48.913226 (+ 2102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:48.916947 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:48.916954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:48.917010 (+ 56us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:48.917644 (+ 634us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:48.918080 (+ 436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:48.918083 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:48.918086 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:48.918287 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:48.918886 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:48.918893 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:48.919309 (+ 416us) client_negotiation.cc:769] Sending connection context
0925 02:48:48.919579 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 02:48:48.920014 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":354,"threads_started":1}
I20260925 02:48:48.922838 24289 heartbeater.cc:347] Connected to a master server at 127.5.56.190:36407
I20260925 02:48:48.923166 24289 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:48.924019 24289 heartbeater.cc:511] Master 127.5.56.190:36407 requested a full tablet report, sending...
I20260925 02:48:48.925467 23827 ts_manager.cc:194] Registered new tserver with Master: d7432997b92a4c6986c67d92b3e31296 (127.5.56.131:36099)
I20260925 02:48:48.926324 23827 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:52901
I20260925 02:48:48.930425 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:48:48.948381 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.934700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.5.56.190:36407)
0925 02:48:48.934890 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:48.934895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:48.935358 (+ 463us) server_negotiation.cc:407] Connection header received
0925 02:48:48.935513 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:48.935517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:48.935583 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:48.935663 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:48.937192 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.937951 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.939491 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.939973 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.940075 (+ 102us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.942897 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:48.942901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:48.942904 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:48.942936 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:48.945373 (+ 2437us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.946490 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.946494 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.946496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.946623 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:48.947069 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:48.947072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:48.947074 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:48.947396 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:48.947608 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:48.947979 (+ 371us) server_negotiation.cc:299] Negotiation successful
0925 02:48:48.948140 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:48:48.952652 23827 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:48.960080 23827 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:48.983341 24302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.972832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:36099 (local address 127.0.0.1:41938)
0925 02:48:48.973643 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:48.973686 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:48.973850 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:48.974925 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:48.974934 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:48.974968 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:48.975356 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.975371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.977929 (+ 2558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.977936 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.980624 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.980640 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.982112 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.982118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.982444 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.982512 (+ 68us) client_negotiation.cc:769] Sending connection context
0925 02:48:48.982648 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 02:48:48.982883 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":120,"thread_start_us":423,"threads_started":1}
I20260925 02:48:48.983363 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.974363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.5.56.129:44667)
0925 02:48:48.975404 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:48.975413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:48.975441 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:48:48.975537 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:48.975544 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:48.975798 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:48.976625 (+ 827us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:48.976886 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.978150 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.980827 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.982164 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.982328 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.982598 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:48.982881 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:48:48.983046 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":853,"thread_start_us":670,"threads_started":1}
I20260925 02:48:48.983417 24304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.974064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:44667 (local address 127.0.0.1:46574)
0925 02:48:48.974697 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:48.974715 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:48.974822 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:48.976051 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:48.976059 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:48.976088 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:48.976440 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.976448 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.978312 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.978321 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.980638 (+ 2317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.980655 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.982459 (+ 1804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.982465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.982679 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.982707 (+ 28us) client_negotiation.cc:769] Sending connection context
0925 02:48:48.982821 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:48:48.982959 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"mutex_wait_us":35,"thread_start_us":354,"threads_started":1}
I20260925 02:48:48.984344 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.973106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.5.56.131:36099)
0925 02:48:48.973831 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:48.973838 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:48.973866 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:48:48.974082 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:48.974089 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:48.974366 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:48.974596 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:48.975561 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.977766 (+ 2205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.980825 (+ 3059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.981890 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.982027 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.982272 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:48.983811 (+ 1539us) server_negotiation.cc:299] Negotiation successful
0925 02:48:48.983975 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":576,"thread_start_us":425,"threads_started":1}
I20260925 02:48:48.987155 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.975062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49114 (local address 127.5.56.130:35971)
0925 02:48:48.978734 (+ 3672us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:48.978743 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:48.978773 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:48:48.978862 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:48.978869 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:48.979092 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:48.979284 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:48.979969 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.981507 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.984926 (+ 3419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.986067 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.986235 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.986467 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:48.986680 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 02:48:48.986841 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3493,"thread_start_us":3390,"threads_started":1}
I20260925 02:48:48.987180 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:48.975447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:35971 (local address 127.0.0.1:49114)
0925 02:48:48.976023 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:48.976039 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:48.976126 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:48.979307 (+ 3181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:48.979314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:48.979344 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:48.979745 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.979756 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.981669 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.981675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.984735 (+ 3060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:48.984750 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:48.986280 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:48.986286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:48.986477 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:48.986518 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 02:48:48.986628 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 02:48:48.986772 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":1,"spinlock_wait_cycles":7552,"thread_start_us":404,"threads_started":1}
I20260925 02:48:48.988109 24223 tablet_service.cc:1467] Processing CreateTablet for tablet 5aef5defb4ee45e6940a6214aa04a732 (DEFAULT_TABLE table=test-table [id=9d4a6dbbc00948148ecd387f61adce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:48.988000 23953 tablet_service.cc:1467] Processing CreateTablet for tablet 5aef5defb4ee45e6940a6214aa04a732 (DEFAULT_TABLE table=test-table [id=9d4a6dbbc00948148ecd387f61adce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:48.989620 23953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aef5defb4ee45e6940a6214aa04a732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:48.989639 24223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aef5defb4ee45e6940a6214aa04a732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:48.992486 24088 tablet_service.cc:1467] Processing CreateTablet for tablet 5aef5defb4ee45e6940a6214aa04a732 (DEFAULT_TABLE table=test-table [id=9d4a6dbbc00948148ecd387f61adce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:48.994001 24088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aef5defb4ee45e6940a6214aa04a732. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:49.005187 24308 tablet_bootstrap.cc:492] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: Bootstrap starting.
I20260925 02:48:49.006258 24309 tablet_bootstrap.cc:492] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: Bootstrap starting.
I20260925 02:48:49.007222 24310 tablet_bootstrap.cc:492] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: Bootstrap starting.
I20260925 02:48:49.010861 24308 tablet_bootstrap.cc:654] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:49.010982 24310 tablet_bootstrap.cc:654] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:49.011035 24309 tablet_bootstrap.cc:654] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:49.011996 24308 log.cc:824] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:49.012220 24309 log.cc:824] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:49.012220 24310 log.cc:824] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:49.015532 24310 tablet_bootstrap.cc:492] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: No bootstrap required, opened a new log
I20260925 02:48:49.015582 24308 tablet_bootstrap.cc:492] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: No bootstrap required, opened a new log
I20260925 02:48:49.015865 24308 ts_tablet_manager.cc:1397] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20260925 02:48:49.015870 24310 ts_tablet_manager.cc:1397] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260925 02:48:49.016065 24309 tablet_bootstrap.cc:492] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: No bootstrap required, opened a new log
I20260925 02:48:49.016321 24309 ts_tablet_manager.cc:1397] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260925 02:48:49.024168 24310 raft_consensus.cc:348] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7432997b92a4c6986c67d92b3e31296" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36099 } } peers { permanent_uuid: "cb553106db7948abb85cbf5451fd900c" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44667 } } peers { permanent_uuid: "af1a0b298e9141bbad0fc65f56a463fa" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35971 } }
I20260925 02:48:49.024612 24310 raft_consensus.cc:374] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:49.024761 24310 raft_consensus.cc:729] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1a0b298e9141bbad0fc65f56a463fa, State: Initialized, Role: FOLLOWER
I20260925 02:48:49.025453 24310 consensus_queue.cc:260] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7432997b92a4c6986c67d92b3e31296" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36099 } } peers { permanent_uuid: "cb553106db7948abb85cbf5451fd900c" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44667 } } peers { permanent_uuid: "af1a0b298e9141bbad0fc65f56a463fa" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35971 } }
I20260925 02:48:49.026747 24153 heartbeater.cc:503] Master 127.5.56.190:36407 was elected leader, sending a full tablet report...
I20260925 02:48:49.027485 24310 ts_tablet_manager.cc:1428] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260925 02:48:49.027212 24309 raft_consensus.cc:348] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7432997b92a4c6986c67d92b3e31296" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36099 } } peers { permanent_uuid: "cb553106db7948abb85cbf5451fd900c" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44667 } } peers { permanent_uuid: "af1a0b298e9141bbad0fc65f56a463fa" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35971 } }
I20260925 02:48:49.027212 24308 raft_consensus.cc:348] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7432997b92a4c6986c67d92b3e31296" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36099 } } peers { permanent_uuid: "cb553106db7948abb85cbf5451fd900c" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44667 } } peers { permanent_uuid: "af1a0b298e9141bbad0fc65f56a463fa" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35971 } }
I20260925 02:48:49.027669 24309 raft_consensus.cc:374] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:49.027662 24308 raft_consensus.cc:374] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:49.027769 24309 raft_consensus.cc:729] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7432997b92a4c6986c67d92b3e31296, State: Initialized, Role: FOLLOWER
I20260925 02:48:49.027768 24308 raft_consensus.cc:729] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb553106db7948abb85cbf5451fd900c, State: Initialized, Role: FOLLOWER
I20260925 02:48:49.028429 24308 consensus_queue.cc:260] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7432997b92a4c6986c67d92b3e31296" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36099 } } peers { permanent_uuid: "cb553106db7948abb85cbf5451fd900c" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44667 } } peers { permanent_uuid: "af1a0b298e9141bbad0fc65f56a463fa" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35971 } }
I20260925 02:48:49.028613 24309 consensus_queue.cc:260] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7432997b92a4c6986c67d92b3e31296" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36099 } } peers { permanent_uuid: "cb553106db7948abb85cbf5451fd900c" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44667 } } peers { permanent_uuid: "af1a0b298e9141bbad0fc65f56a463fa" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35971 } }
I20260925 02:48:49.030364 24289 heartbeater.cc:503] Master 127.5.56.190:36407 was elected leader, sending a full tablet report...
I20260925 02:48:49.032420 24308 ts_tablet_manager.cc:1428] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: Time spent starting tablet: real 0.016s user 0.013s sys 0.002s
I20260925 02:48:49.033831 24309 ts_tablet_manager.cc:1428] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: Time spent starting tablet: real 0.017s user 0.006s sys 0.009s
W20260925 02:48:49.068933 24019 tablet.cc:2369] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:48:49.111106 24154 tablet.cc:2369] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:48:49.150975 24290 tablet.cc:2369] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:49.198189 24314 raft_consensus.cc:482] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:49.198582 24314 raft_consensus.cc:504] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7432997b92a4c6986c67d92b3e31296" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36099 } } peers { permanent_uuid: "cb553106db7948abb85cbf5451fd900c" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44667 } } peers { permanent_uuid: "af1a0b298e9141bbad0fc65f56a463fa" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35971 } }
I20260925 02:48:49.200297 24314 leader_election.cc:288] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7432997b92a4c6986c67d92b3e31296 (127.5.56.131:36099), cb553106db7948abb85cbf5451fd900c (127.5.56.129:44667)
I20260925 02:48:49.207050 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.200726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:34493 (local address 127.5.56.131:36099)
0925 02:48:49.200950 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.200958 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.201455 (+ 497us) server_negotiation.cc:407] Connection header received
0925 02:48:49.201758 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.201767 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.201887 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.201989 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:49.202652 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.203389 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.205206 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.205999 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.206195 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.206277 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.206626 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.206754 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:48:49.207193 24317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.200721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:36099 (local address 127.5.56.130:34493)
0925 02:48:49.201336 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:49.201353 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:49.201558 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:49.202032 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:49.202036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:49.202072 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:49.202501 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.202511 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.203530 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.203535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.205041 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.205055 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.206225 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.206232 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.206460 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.206492 (+ 32us) client_negotiation.cc:769] Sending connection context
0925 02:48:49.206599 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:48:49.206748 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":493,"mutex_wait_us":1,"thread_start_us":335,"threads_started":1}
I20260925 02:48:49.207486 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.201811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:38489 (local address 127.5.56.129:44667)
0925 02:48:49.202034 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.202041 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.202224 (+ 183us) server_negotiation.cc:407] Connection header received
0925 02:48:49.202459 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.202464 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.202531 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.202629 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:49.203166 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.203882 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.205613 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.206508 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.206678 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.206768 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.207064 (+ 296us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.207198 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:48:49.207599 24318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.201506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:44667 (local address 127.5.56.130:38489)
0925 02:48:49.202126 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:49.202142 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:49.202302 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:49.202646 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:49.202649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:49.202671 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:49.203009 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.203016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.204023 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.204028 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.205457 (+ 1429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:49.205470 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.206724 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.206730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:49.206905 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.206928 (+ 23us) client_negotiation.cc:769] Sending connection context
0925 02:48:49.207031 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 02:48:49.207161 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":307,"threads_started":1}
I20260925 02:48:49.208496 24243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aef5defb4ee45e6940a6214aa04a732" candidate_uuid: "af1a0b298e9141bbad0fc65f56a463fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7432997b92a4c6986c67d92b3e31296" is_pre_election: true
I20260925 02:48:49.208633 23973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aef5defb4ee45e6940a6214aa04a732" candidate_uuid: "af1a0b298e9141bbad0fc65f56a463fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb553106db7948abb85cbf5451fd900c" is_pre_election: true
I20260925 02:48:49.209081 24243 raft_consensus.cc:2454] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af1a0b298e9141bbad0fc65f56a463fa in term 0.
I20260925 02:48:49.209174 23973 raft_consensus.cc:2454] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af1a0b298e9141bbad0fc65f56a463fa in term 0.
I20260925 02:48:49.209923 24040 leader_election.cc:302] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af1a0b298e9141bbad0fc65f56a463fa, d7432997b92a4c6986c67d92b3e31296; no voters:
I20260925 02:48:49.210335 24314 raft_consensus.cc:2781] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:49.210549 24314 raft_consensus.cc:482] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:49.210673 24314 raft_consensus.cc:3037] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:49.212926 24314 raft_consensus.cc:504] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7432997b92a4c6986c67d92b3e31296" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36099 } } peers { permanent_uuid: "cb553106db7948abb85cbf5451fd900c" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44667 } } peers { permanent_uuid: "af1a0b298e9141bbad0fc65f56a463fa" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35971 } }
I20260925 02:48:49.213542 24314 leader_election.cc:288] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [CANDIDATE]: Term 1 election: Requested vote from peers d7432997b92a4c6986c67d92b3e31296 (127.5.56.131:36099), cb553106db7948abb85cbf5451fd900c (127.5.56.129:44667)
I20260925 02:48:49.214133 24243 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aef5defb4ee45e6940a6214aa04a732" candidate_uuid: "af1a0b298e9141bbad0fc65f56a463fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7432997b92a4c6986c67d92b3e31296"
I20260925 02:48:49.214171 23973 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aef5defb4ee45e6940a6214aa04a732" candidate_uuid: "af1a0b298e9141bbad0fc65f56a463fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cb553106db7948abb85cbf5451fd900c"
I20260925 02:48:49.214340 24243 raft_consensus.cc:3037] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:49.214433 23973 raft_consensus.cc:3037] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:49.217007 24243 raft_consensus.cc:2454] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af1a0b298e9141bbad0fc65f56a463fa in term 1.
I20260925 02:48:49.217006 23973 raft_consensus.cc:2454] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af1a0b298e9141bbad0fc65f56a463fa in term 1.
I20260925 02:48:49.217445 24040 leader_election.cc:302] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af1a0b298e9141bbad0fc65f56a463fa, d7432997b92a4c6986c67d92b3e31296; no voters:
I20260925 02:48:49.217785 24314 raft_consensus.cc:2781] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:49.218276 24314 raft_consensus.cc:686] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 1 LEADER]: Becoming Leader. State: Replica: af1a0b298e9141bbad0fc65f56a463fa, State: Running, Role: LEADER
I20260925 02:48:49.218824 24314 consensus_queue.cc:237] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7432997b92a4c6986c67d92b3e31296" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36099 } } peers { permanent_uuid: "cb553106db7948abb85cbf5451fd900c" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44667 } } peers { permanent_uuid: "af1a0b298e9141bbad0fc65f56a463fa" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35971 } }
I20260925 02:48:49.223814 23827 catalog_manager.cc:5495] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa reported cstate change: term changed from 0 to 1, leader changed from <none> to af1a0b298e9141bbad0fc65f56a463fa (127.5.56.130). New cstate: current_term: 1 leader_uuid: "af1a0b298e9141bbad0fc65f56a463fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7432997b92a4c6986c67d92b3e31296" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cb553106db7948abb85cbf5451fd900c" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44667 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af1a0b298e9141bbad0fc65f56a463fa" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 35971 } health_report { overall_health: HEALTHY } } }
I20260925 02:48:49.287367 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.281890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49128 (local address 127.5.56.130:35971)
0925 02:48:49.282113 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.282118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.282137 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:48:49.282296 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.282299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.282460 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.282647 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:49.283079 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.283774 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.285098 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.285515 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.285609 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.286125 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:49.286705 (+ 580us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:49.286859 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.286976 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.287049 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:48:49.296304 23973 raft_consensus.cc:1261] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c [term 1 FOLLOWER]: Refusing update from remote peer af1a0b298e9141bbad0fc65f56a463fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:49.296310 24243 raft_consensus.cc:1261] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296 [term 1 FOLLOWER]: Refusing update from remote peer af1a0b298e9141bbad0fc65f56a463fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:49.297438 24314 consensus_queue.cc:1035] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7432997b92a4c6986c67d92b3e31296" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 36099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:49.297753 24319 consensus_queue.cc:1035] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "cb553106db7948abb85cbf5451fd900c" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44667 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:49.310086 24322 mvcc.cc:205] Tried to move back new op lower bound from 7333087351985999872 to 7333087351684276224. Current Snapshot: MvccSnapshot[applied={T|T < 7333087351985999872}]
I20260925 02:48:49.314055 24323 mvcc.cc:205] Tried to move back new op lower bound from 7333087351985999872 to 7333087351684276224. Current Snapshot: MvccSnapshot[applied={T|T < 7333087351985999872}]
I20260925 02:48:49.327805 23827 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:49.328130 23827 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:49.332260 23827 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 531831f813d94e5eb89298a9abb664ef: Sending DeleteTablet for 3 replicas of tablet 5aef5defb4ee45e6940a6214aa04a732
I20260925 02:48:49.333578 24223 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aef5defb4ee45e6940a6214aa04a732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:49 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:41938
I20260925 02:48:49.333727 24088 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aef5defb4ee45e6940a6214aa04a732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:49 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:49114
I20260925 02:48:49.333873 23953 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aef5defb4ee45e6940a6214aa04a732 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:49 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:46574
I20260925 02:48:49.339003 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 23889
I20260925 02:48:49.346939 24333 tablet_replica.cc:330] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: stopping tablet replica
I20260925 02:48:49.347729 24334 tablet_replica.cc:330] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: stopping tablet replica
I20260925 02:48:49.347865 24332 tablet_replica.cc:330] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: stopping tablet replica
I20260925 02:48:49.348403 24332 raft_consensus.cc:2229] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:49.348398 24333 raft_consensus.cc:2229] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:49.349007 24332 raft_consensus.cc:2258] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:49.348954 24334 raft_consensus.cc:2229] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:49.349524 24334 raft_consensus.cc:2258] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:49.350306 24333 raft_consensus.cc:2258] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:49.351436 24332 ts_tablet_manager.cc:1905] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:49.352564 24333 ts_tablet_manager.cc:1905] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:49.352571 24334 ts_tablet_manager.cc:1905] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:49.359624 24334 ts_tablet_manager.cc:1918] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:49.359854 24334 log.cc:1196] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/5aef5defb4ee45e6940a6214aa04a732
I20260925 02:48:49.360518 24334 ts_tablet_manager.cc:1939] T 5aef5defb4ee45e6940a6214aa04a732 P cb553106db7948abb85cbf5451fd900c: Deleting consensus metadata
I20260925 02:48:49.360589 24333 ts_tablet_manager.cc:1918] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:49.360788 24333 log.cc:1196] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/5aef5defb4ee45e6940a6214aa04a732
I20260925 02:48:49.361485 24333 ts_tablet_manager.cc:1939] T 5aef5defb4ee45e6940a6214aa04a732 P af1a0b298e9141bbad0fc65f56a463fa: Deleting consensus metadata
I20260925 02:48:49.362859 23811 catalog_manager.cc:4841] TS cb553106db7948abb85cbf5451fd900c (127.5.56.129:44667): tablet 5aef5defb4ee45e6940a6214aa04a732 (table test-table [id=9d4a6dbbc00948148ecd387f61adce3a]) successfully deleted
I20260925 02:48:49.363236 23811 catalog_manager.cc:4841] TS af1a0b298e9141bbad0fc65f56a463fa (127.5.56.130:35971): tablet 5aef5defb4ee45e6940a6214aa04a732 (table test-table [id=9d4a6dbbc00948148ecd387f61adce3a]) successfully deleted
Sep 25 02:48:49 dist-test-slave-n563 krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:48:49.368363 24332 ts_tablet_manager.cc:1918] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:49.368666 24332 log.cc:1196] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/5aef5defb4ee45e6940a6214aa04a732
I20260925 02:48:49.369287 24332 ts_tablet_manager.cc:1939] T 5aef5defb4ee45e6940a6214aa04a732 P d7432997b92a4c6986c67d92b3e31296: Deleting consensus metadata
I20260925 02:48:49.371461 23811 catalog_manager.cc:4841] TS d7432997b92a4c6986c67d92b3e31296 (127.5.56.131:36099): tablet 5aef5defb4ee45e6940a6214aa04a732 (table test-table [id=9d4a6dbbc00948148ecd387f61adce3a]) successfully deleted
I20260925 02:48:49.373083 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.340816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.5.56.129:44667)
0925 02:48:49.341068 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.341079 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.342298 (+ 1219us) server_negotiation.cc:407] Connection header received
0925 02:48:49.344475 (+ 2177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.344484 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.344633 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.344780 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:49.348703 (+ 3923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.349933 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.351864 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.352527 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.352640 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.365229 (+ 12589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:49.365247 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:49.365269 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:49.365315 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:49.369462 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:49.370607 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:49.370617 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:49.370620 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:49.370823 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:49.371291 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:49.371297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:49.371299 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:49.371824 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:49.372135 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.372470 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.372697 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260925 02:48:49.373782 24013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:49.468626 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 23889
I20260925 02:48:49.488487 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 24024
Sep 25 02:48:49 dist-test-slave-n563 krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:48:49.506454 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.489367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.5.56.130:35971)
0925 02:48:49.489550 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.489558 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.489581 (+ 23us) server_negotiation.cc:407] Connection header received
0925 02:48:49.489825 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.489833 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.489908 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.490026 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:49.491456 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.492293 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.494296 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.494831 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.494936 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.500114 (+ 5178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:49.500130 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:49.500146 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:49.500188 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:49.503366 (+ 3178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:49.504144 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:49.504151 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:49.504153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:49.504327 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:49.504719 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:49.504725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:49.504728 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:49.505122 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:49.505502 (+ 380us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.505786 (+ 284us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.506015 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:48:49.507215 24148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:49.624279 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 24024
I20260925 02:48:49.644238 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 24159
Sep 25 02:48:49 dist-test-slave-n563 krb5kdc[23782](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:48:49.659369 24303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:49.645055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.5.56.131:36099)
0925 02:48:49.645228 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:49.645232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:49.645249 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:48:49.645460 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:49.645465 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:49.645538 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:49.645633 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:49.647022 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.647869 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.649136 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:49.649521 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:49.649616 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:49.654133 (+ 4517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:49.654148 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:49.654164 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:49.654200 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:49.656850 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:49.657508 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:49.657514 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:49.657517 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:49.657687 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:49.658022 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:49.658025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:49.658027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:49.658375 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:49.658660 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:49.658933 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 02:48:49.659094 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:48:49.660041 24283 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:49.750672 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 24159
I20260925 02:48:49.768608 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 23795
I20260925 02:48:49.769737 23858 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:49.882560 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 23795
2026-09-25T02:48:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4579 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T02:48:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:49Z Disabled control of system clock
I20260925 02:48:49.965941 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:34469
--webserver_interface=127.5.56.190
--webserver_port=0
--builtin_ntp_servers=127.5.56.148:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:34469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:48:50.314039 24343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:50.314402 24343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:50.314572 24343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:50.322041 24343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:50.322161 24343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:50.322252 24343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:50.322376 24343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:50.322499 24343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:50.332166 24343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:34469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:34469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:50.333606 24343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:50.335536 24343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:50.348222 24350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:50.348341 24348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:50.348379 24352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:50.349201 24343 server_base.cc:1034] running on GCE node
I20260925 02:48:50.350155 24343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:50.351848 24343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:50.353060 24343 hybrid_clock.cc:648] HybridClock initialized: now 1790304530353020 us; error 49 us; skew 500 ppm
I20260925 02:48:50.353449 24343 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:50.356103 24343 webserver.cc:460] Webserver started at http://127.5.56.190:36353/ using document root <none> and password file <none>
I20260925 02:48:50.356968 24343 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:50.357059 24343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:50.357391 24343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:50.360452 24343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "a0ba076d56f14b878fc1dc74ce245ef5"
format_stamp: "Formatted at 2026-09-25 02:48:50 on dist-test-slave-n563"
server_key: "3e24b36dac3858a020c44dafc4b7418e"
server_key_iv: "fbdef0c1e0294696313d02206dfd27d5"
server_key_version: "encryptionkey@0"
I20260925 02:48:50.361231 24343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "a0ba076d56f14b878fc1dc74ce245ef5"
format_stamp: "Formatted at 2026-09-25 02:48:50 on dist-test-slave-n563"
server_key: "3e24b36dac3858a020c44dafc4b7418e"
server_key_iv: "fbdef0c1e0294696313d02206dfd27d5"
server_key_version: "encryptionkey@0"
I20260925 02:48:50.366714 24343 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260925 02:48:50.370159 24358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:50.371668 24343 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:48:50.371867 24343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "a0ba076d56f14b878fc1dc74ce245ef5"
format_stamp: "Formatted at 2026-09-25 02:48:50 on dist-test-slave-n563"
server_key: "3e24b36dac3858a020c44dafc4b7418e"
server_key_iv: "fbdef0c1e0294696313d02206dfd27d5"
server_key_version: "encryptionkey@0"
I20260925 02:48:50.372112 24343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:50.388657 24343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:50.389595 24343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:50.389933 24343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:50.410050 24343 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:34469
I20260925 02:48:50.410071 24409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:34469 every 8 connection(s)
I20260925 02:48:50.411569 24343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:50.412731 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 24343
I20260925 02:48:50.412918 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:48:50.413491 5346 external_mini_cluster.cc:1428] Setting key 140e99478612728a0aee6785ee9d6ba4
I20260925 02:48:50.416069 24410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:50.431437 24410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5: Bootstrap starting.
I20260925 02:48:50.431739 24413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:50.415788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.5.56.190:34469)
0925 02:48:50.417019 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:50.417034 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:50.419043 (+ 2009us) server_negotiation.cc:407] Connection header received
0925 02:48:50.420975 (+ 1932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:50.421009 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:50.421668 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:50.422133 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:50.422752 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:50.424805 (+ 2053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:50.426286 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:50.427387 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:50.427578 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:50.428741 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:50.428752 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:50.428773 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:50.429596 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:50.429696 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:50.429723 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:50.429839 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:50.430047 (+ 208us) server_negotiation.cc:299] Negotiation successful
0925 02:48:50.430424 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":437,"threads_started":1}
I20260925 02:48:50.434830 24410 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:50.435817 24410 log.cc:824] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:50.438617 24410 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5: No bootstrap required, opened a new log
I20260925 02:48:50.445848 24410 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ba076d56f14b878fc1dc74ce245ef5" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34469 } }
I20260925 02:48:50.446271 24410 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:50.446444 24410 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0ba076d56f14b878fc1dc74ce245ef5, State: Initialized, Role: FOLLOWER
I20260925 02:48:50.447032 24410 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ba076d56f14b878fc1dc74ce245ef5" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34469 } }
I20260925 02:48:50.447288 24410 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:50.447433 24410 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:50.447611 24410 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:50.449965 24410 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ba076d56f14b878fc1dc74ce245ef5" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34469 } }
I20260925 02:48:50.450547 24410 leader_election.cc:302] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0ba076d56f14b878fc1dc74ce245ef5; no voters:
I20260925 02:48:50.451207 24410 leader_election.cc:288] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:50.451380 24415 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:50.452211 24415 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [term 1 LEADER]: Becoming Leader. State: Replica: a0ba076d56f14b878fc1dc74ce245ef5, State: Running, Role: LEADER
I20260925 02:48:50.452715 24415 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ba076d56f14b878fc1dc74ce245ef5" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34469 } }
I20260925 02:48:50.453598 24410 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:50.455977 24417 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0ba076d56f14b878fc1dc74ce245ef5. Latest consensus state: current_term: 1 leader_uuid: "a0ba076d56f14b878fc1dc74ce245ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ba076d56f14b878fc1dc74ce245ef5" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34469 } } }
I20260925 02:48:50.456482 24417 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:50.457808 24423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:50.457712 24416 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0ba076d56f14b878fc1dc74ce245ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0ba076d56f14b878fc1dc74ce245ef5" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34469 } } }
I20260925 02:48:50.458106 24416 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:50.463529 24423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:50.472936 24423 catalog_manager.cc:1347] Generated new cluster ID: 59c37dd0463d467f807e0c10f2f5f4f0
I20260925 02:48:50.473081 24423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:50.484946 24423 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:50.486281 24423 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:50.495046 24423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5: Generated new TSK 0
I20260925 02:48:50.495946 24423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:48:50.507377 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34469
--builtin_ntp_servers=127.5.56.148:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:48:50.854166 24434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:50.854564 24434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:50.854743 24434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:50.862471 24434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:50.862586 24434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:50.862768 24434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:48:50.872641 24434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:50.874051 24434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:50.876008 24434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:50.889760 24439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:50.889688 24440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:50.889722 24442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:50.890609 24434 server_base.cc:1034] running on GCE node
I20260925 02:48:50.891240 24434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:50.892201 24434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:50.893412 24434 hybrid_clock.cc:648] HybridClock initialized: now 1790304530893359 us; error 41 us; skew 500 ppm
I20260925 02:48:50.893697 24434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:50.896111 24434 webserver.cc:460] Webserver started at http://127.5.56.129:40109/ using document root <none> and password file <none>
I20260925 02:48:50.896781 24434 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:50.896878 24434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:50.897156 24434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:50.900197 24434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "90795bb485344ebba300a00c516ac67a"
format_stamp: "Formatted at 2026-09-25 02:48:50 on dist-test-slave-n563"
server_key: "51bc5f3df129b6244b82b06b5916fbb7"
server_key_iv: "67d401178a217aad57f84465f9a0155f"
server_key_version: "encryptionkey@0"
I20260925 02:48:50.900977 24434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "90795bb485344ebba300a00c516ac67a"
format_stamp: "Formatted at 2026-09-25 02:48:50 on dist-test-slave-n563"
server_key: "51bc5f3df129b6244b82b06b5916fbb7"
server_key_iv: "67d401178a217aad57f84465f9a0155f"
server_key_version: "encryptionkey@0"
I20260925 02:48:50.906340 24434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 02:48:50.909945 24448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:50.911419 24434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:50.911569 24434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "90795bb485344ebba300a00c516ac67a"
format_stamp: "Formatted at 2026-09-25 02:48:50 on dist-test-slave-n563"
server_key: "51bc5f3df129b6244b82b06b5916fbb7"
server_key_iv: "67d401178a217aad57f84465f9a0155f"
server_key_version: "encryptionkey@0"
I20260925 02:48:50.911770 24434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:50.929530 24434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:50.930464 24434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:50.930801 24434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:50.931823 24434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:50.933363 24434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:50.933494 24434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:50.933634 24434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:50.933722 24434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:50.968255 24434 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:44559
I20260925 02:48:50.968292 24562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:44559 every 8 connection(s)
I20260925 02:48:50.969672 24434 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:50.977861 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 24434
I20260925 02:48:50.978089 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:50.978602 5346 external_mini_cluster.cc:1428] Setting key 7b967517db039c0e61a89a41733cd19d
I20260925 02:48:50.981379 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34469
--builtin_ntp_servers=127.5.56.148:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:48:50.983196 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:50.972502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:33355 (local address 127.5.56.190:34469)
0925 02:48:50.973018 (+ 516us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:50.973022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:50.974076 (+ 1054us) server_negotiation.cc:407] Connection header received
0925 02:48:50.975986 (+ 1910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:50.975991 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:50.976094 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:50.976236 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:50.977377 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:50.978198 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:50.980039 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:50.980655 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:50.980811 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:50.982286 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:50.982292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:50.982297 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:50.982327 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:50.982367 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:50.982417 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:50.982505 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:50.982754 (+ 249us) server_negotiation.cc:299] Negotiation successful
0925 02:48:50.982860 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":301,"threads_started":1}
I20260925 02:48:50.984297 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:50.972988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:34469 (local address 127.5.56.129:33355)
0925 02:48:50.973911 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:50.973946 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:50.975703 (+ 1757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:50.976483 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:50.976491 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:50.976582 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:50.977204 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:50.977222 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:50.978361 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:50.978373 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:50.979889 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:50.979902 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:50.980920 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:50.980927 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:50.981183 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:50.981948 (+ 765us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:50.981951 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:50.981995 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:50.982079 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:50.982502 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:50.982516 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:50.982527 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:48:50.982724 (+ 197us) client_negotiation.cc:240] Negotiation successful
0925 02:48:50.983164 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":291,"threads_started":1}
I20260925 02:48:50.985915 24563 heartbeater.cc:347] Connected to a master server at 127.5.56.190:34469
I20260925 02:48:50.986244 24563 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:50.987110 24563 heartbeater.cc:511] Master 127.5.56.190:34469 requested a full tablet report, sending...
I20260925 02:48:50.989377 24375 ts_manager.cc:194] Registered new tserver with Master: 90795bb485344ebba300a00c516ac67a (127.5.56.129:44559)
I20260925 02:48:50.990995 24375 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.129:33355
W20260925 02:48:51.324136 24567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:51.324489 24567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:51.324673 24567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:51.332073 24567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:51.332252 24567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:51.332430 24567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:48:51.341710 24567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:51.343247 24567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:51.345108 24567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:51.358901 24572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:51.358981 24575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:51.358937 24573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:51.359992 24567 server_base.cc:1034] running on GCE node
I20260925 02:48:51.360704 24567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:51.361716 24567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:51.362955 24567 hybrid_clock.cc:648] HybridClock initialized: now 1790304531362896 us; error 59 us; skew 500 ppm
I20260925 02:48:51.363344 24567 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:51.365731 24567 webserver.cc:460] Webserver started at http://127.5.56.130:43597/ using document root <none> and password file <none>
I20260925 02:48:51.366560 24567 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:51.366690 24567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:51.366990 24567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:51.369884 24567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "501f57d1365c47909e7fa077a141961f"
format_stamp: "Formatted at 2026-09-25 02:48:51 on dist-test-slave-n563"
server_key: "e3b498b7d3155744c37ab2d8497296db"
server_key_iv: "36ccededf221d78a6b7f82d4a61fc7a6"
server_key_version: "encryptionkey@0"
I20260925 02:48:51.370730 24567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "501f57d1365c47909e7fa077a141961f"
format_stamp: "Formatted at 2026-09-25 02:48:51 on dist-test-slave-n563"
server_key: "e3b498b7d3155744c37ab2d8497296db"
server_key_iv: "36ccededf221d78a6b7f82d4a61fc7a6"
server_key_version: "encryptionkey@0"
I20260925 02:48:51.375978 24567 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:48:51.379827 24581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:51.381263 24567 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:48:51.381405 24567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "501f57d1365c47909e7fa077a141961f"
format_stamp: "Formatted at 2026-09-25 02:48:51 on dist-test-slave-n563"
server_key: "e3b498b7d3155744c37ab2d8497296db"
server_key_iv: "36ccededf221d78a6b7f82d4a61fc7a6"
server_key_version: "encryptionkey@0"
I20260925 02:48:51.381595 24567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:51.406802 24567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:51.407899 24567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:51.408351 24567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:51.410002 24567 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:51.412030 24567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:51.412146 24567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:51.412263 24567 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:51.412345 24567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:51.447140 24567 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:41623
I20260925 02:48:51.447247 24694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:41623 every 8 connection(s)
I20260925 02:48:51.448527 24567 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:51.458636 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 24567
I20260925 02:48:51.458850 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:51.459446 5346 external_mini_cluster.cc:1428] Setting key c99eb29df93f7d6ee95098f26358bcf1
I20260925 02:48:51.462273 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:51.451219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:39725 (local address 127.5.56.190:34469)
0925 02:48:51.451445 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:51.451452 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:51.453119 (+ 1667us) server_negotiation.cc:407] Connection header received
0925 02:48:51.455205 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:51.455209 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:51.455285 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:51.455378 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:51.456477 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.457206 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.459042 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.459659 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.459798 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:51.461460 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:51.461465 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:51.461469 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:51.461487 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:51.461520 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:51.461532 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:51.461629 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:51.461864 (+ 235us) server_negotiation.cc:299] Negotiation successful
0925 02:48:51.461963 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 02:48:51.462677 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34469
--builtin_ntp_servers=127.5.56.148:39915
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:48:51.463500 24697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:51.451982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:34469 (local address 127.5.56.130:39725)
0925 02:48:51.452952 (+ 970us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:51.452999 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:51.454886 (+ 1887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:51.455583 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:51.455591 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:51.455734 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:51.456320 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:51.456339 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.457360 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.457364 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:51.458859 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:51.458873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.460004 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.460011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:51.460354 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:51.461122 (+ 768us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:51.461125 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:51.461166 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:51.461257 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:51.461629 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:51.461643 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:51.461654 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 02:48:51.461855 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 02:48:51.462286 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":291,"threads_started":1}
I20260925 02:48:51.465106 24695 heartbeater.cc:347] Connected to a master server at 127.5.56.190:34469
I20260925 02:48:51.465430 24695 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:51.466207 24695 heartbeater.cc:511] Master 127.5.56.190:34469 requested a full tablet report, sending...
I20260925 02:48:51.467836 24375 ts_manager.cc:194] Registered new tserver with Master: 501f57d1365c47909e7fa077a141961f (127.5.56.130:41623)
I20260925 02:48:51.469075 24375 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.130:39725
W20260925 02:48:51.803257 24698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:51.803629 24698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:51.803802 24698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:51.811233 24698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:51.811406 24698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:51.811596 24698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:48:51.821065 24698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39915
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:51.822463 24698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:51.824277 24698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:51.840771 24704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:51.840762 24707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:51.840771 24705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:51.841560 24698 server_base.cc:1034] running on GCE node
I20260925 02:48:51.842305 24698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:51.843329 24698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:51.844570 24698 hybrid_clock.cc:648] HybridClock initialized: now 1790304531844519 us; error 46 us; skew 500 ppm
I20260925 02:48:51.844914 24698 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:51.847287 24698 webserver.cc:460] Webserver started at http://127.5.56.131:45769/ using document root <none> and password file <none>
I20260925 02:48:51.848024 24698 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:51.848158 24698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:51.848497 24698 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:51.851539 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "8a0ff573c0c34998ac077c553920f8d4"
format_stamp: "Formatted at 2026-09-25 02:48:51 on dist-test-slave-n563"
server_key: "fba45a841a33bfc660fba4b986ccf505"
server_key_iv: "1972d0865dba9f031adc363a89ffa9cf"
server_key_version: "encryptionkey@0"
I20260925 02:48:51.852353 24698 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "8a0ff573c0c34998ac077c553920f8d4"
format_stamp: "Formatted at 2026-09-25 02:48:51 on dist-test-slave-n563"
server_key: "fba45a841a33bfc660fba4b986ccf505"
server_key_iv: "1972d0865dba9f031adc363a89ffa9cf"
server_key_version: "encryptionkey@0"
I20260925 02:48:51.857637 24698 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:48:51.861377 24713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:51.862874 24698 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260925 02:48:51.863057 24698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "8a0ff573c0c34998ac077c553920f8d4"
format_stamp: "Formatted at 2026-09-25 02:48:51 on dist-test-slave-n563"
server_key: "fba45a841a33bfc660fba4b986ccf505"
server_key_iv: "1972d0865dba9f031adc363a89ffa9cf"
server_key_version: "encryptionkey@0"
I20260925 02:48:51.863288 24698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:51.879256 24698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:51.880146 24698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:51.880455 24698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:51.881464 24698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:51.883271 24698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:51.883368 24698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:51.883539 24698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:51.883625 24698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:51.917621 24698 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:38443
I20260925 02:48:51.917675 24825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:38443 every 8 connection(s)
I20260925 02:48:51.919085 24698 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:51.920596 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 24698
I20260925 02:48:51.920830 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:51.921288 5346 external_mini_cluster.cc:1428] Setting key d18e70ae301995ec4ad18e93ace6df2f
I20260925 02:48:51.932286 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:51.921924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:38425 (local address 127.5.56.190:34469)
0925 02:48:51.922123 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:51.922129 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:51.923758 (+ 1629us) server_negotiation.cc:407] Connection header received
0925 02:48:51.925702 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:51.925707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:51.925796 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:51.925916 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:51.927093 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.927838 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.929427 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.929859 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.929996 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:51.931413 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:51.931419 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:51.931430 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:51.931453 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:51.931486 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:51.931498 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:51.931594 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:51.931837 (+ 243us) server_negotiation.cc:299] Negotiation successful
0925 02:48:51.931943 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:48:51.933393 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:51.922562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:34469 (local address 127.5.56.131:38425)
0925 02:48:51.923610 (+ 1048us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:51.923653 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:51.925439 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:51.926160 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:51.926169 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:51.926258 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:51.926926 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:51.926944 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.927997 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.928010 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:51.929261 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:51.929274 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.930091 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.930099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:51.930347 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:51.931085 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:51.931088 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:51.931130 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:51.931213 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:51.931592 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:51.931606 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:51.931616 (+ 10us) client_negotiation.cc:769] Sending connection context
0925 02:48:51.931812 (+ 196us) client_negotiation.cc:240] Negotiation successful
0925 02:48:51.932228 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":328,"threads_started":1}
I20260925 02:48:51.934895 24826 heartbeater.cc:347] Connected to a master server at 127.5.56.190:34469
I20260925 02:48:51.935210 24826 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:51.936043 24826 heartbeater.cc:511] Master 127.5.56.190:34469 requested a full tablet report, sending...
I20260925 02:48:51.937433 24375 ts_manager.cc:194] Registered new tserver with Master: 8a0ff573c0c34998ac077c553920f8d4 (127.5.56.131:38443)
I20260925 02:48:51.938205 24375 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.131:38425
I20260925 02:48:51.949359 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:48:51.960350 24565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:51.953821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49870 (local address 127.5.56.190:34469)
0925 02:48:51.954017 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:51.954023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:51.954598 (+ 575us) server_negotiation.cc:407] Connection header received
0925 02:48:51.954733 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:51.954737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:51.954841 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:51.954984 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:51.955498 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.956229 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.957612 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.958054 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.958194 (+ 140us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:51.959504 (+ 1310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:51.959512 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:51.959522 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:51.959541 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:51.959571 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:51.959583 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:51.959739 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:51.959959 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 02:48:51.960019 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:48:51.964305 24375 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49870:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:51.971414 24375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:51.993997 24563 heartbeater.cc:503] Master 127.5.56.190:34469 was elected leader, sending a full tablet report...
I20260925 02:48:52.001453 24842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:51.983794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:41623 (local address 127.0.0.1:57690)
0925 02:48:51.991876 (+ 8082us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:51.991901 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:51.992045 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:51.993279 (+ 1234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:51.993287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:51.993318 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:51.993754 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:51.993780 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.995707 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.995715 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:51.998148 (+ 2433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:51.998163 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.000458 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.000464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:52.000677 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.000733 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 02:48:52.000877 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 02:48:52.001077 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7909,"mutex_wait_us":3,"thread_start_us":7740,"threads_started":1}
I20260925 02:48:52.001472 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:51.984424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57690 (local address 127.5.56.130:41623)
0925 02:48:51.985097 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:51.985111 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:51.992205 (+ 7094us) server_negotiation.cc:407] Connection header received
0925 02:48:51.992333 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:51.992344 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:51.992660 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:51.992885 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:51.993986 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.995538 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.998424 (+ 2886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.999909 (+ 1485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.000049 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.000333 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:52.000957 (+ 624us) server_negotiation.cc:299] Negotiation successful
0925 02:48:52.001141 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":543,"threads_started":1}
I20260925 02:48:52.003446 24843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:51.986637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:38443 (local address 127.0.0.1:36702)
0925 02:48:51.992081 (+ 5444us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:51.992098 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:51.992193 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:51.995302 (+ 3109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:51.995308 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:51.995333 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:51.995777 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:51.995790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.998019 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:51.998026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:52.000193 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:52.000210 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.002196 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.002203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:52.002595 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.002776 (+ 181us) client_negotiation.cc:769] Sending connection context
0925 02:48:52.002872 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 02:48:52.003010 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5323,"mutex_wait_us":150,"thread_start_us":6428,"threads_started":1}
I20260925 02:48:52.004205 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:51.986961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36702 (local address 127.5.56.131:38443)
0925 02:48:51.994685 (+ 7724us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:51.994694 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:51.994728 (+ 34us) server_negotiation.cc:407] Connection header received
0925 02:48:51.994818 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:51.994826 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:51.995075 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:51.995997 (+ 922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:51.996279 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:51.997852 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.000473 (+ 2621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.001991 (+ 1518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.002146 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.002451 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:52.003676 (+ 1225us) server_negotiation.cc:299] Negotiation successful
0925 02:48:52.003853 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7542,"thread_start_us":6668,"threads_started":1}
I20260925 02:48:52.006779 24839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:51.983684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:44559 (local address 127.0.0.1:39652)
0925 02:48:51.987289 (+ 3605us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:51.987338 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:51.987566 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:51.999740 (+ 12174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:51.999745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:51.999771 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:52.000260 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:52.000274 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.001951 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.001959 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:52.004312 (+ 2353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:52.004329 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.005935 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.005941 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:52.006110 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.006153 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 02:48:52.006253 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 02:48:52.006427 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3274,"mutex_wait_us":3,"thread_start_us":1729,"threads_started":1}
I20260925 02:48:52.006836 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:51.983910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39652 (local address 127.5.56.129:44559)
0925 02:48:51.999223 (+ 15313us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:51.999229 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:51.999273 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:48:51.999350 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:51.999355 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:51.999546 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:51.999752 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:52.000452 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.001804 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.004537 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.005731 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.005902 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.006167 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:52.006295 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 02:48:52.006474 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15154,"thread_start_us":16069,"threads_started":1}
I20260925 02:48:52.006233 24630 tablet_service.cc:1467] Processing CreateTablet for tablet cba26d389d594230bff5a285090d81fa (DEFAULT_TABLE table=test-table [id=96abb60105c54c56994bda84bd37b007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:52.007812 24630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cba26d389d594230bff5a285090d81fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:52.007920 24761 tablet_service.cc:1467] Processing CreateTablet for tablet cba26d389d594230bff5a285090d81fa (DEFAULT_TABLE table=test-table [id=96abb60105c54c56994bda84bd37b007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:52.010097 24761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cba26d389d594230bff5a285090d81fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:52.010330 24496 tablet_service.cc:1467] Processing CreateTablet for tablet cba26d389d594230bff5a285090d81fa (DEFAULT_TABLE table=test-table [id=96abb60105c54c56994bda84bd37b007]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:52.011503 24496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cba26d389d594230bff5a285090d81fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:52.022554 24845 tablet_bootstrap.cc:492] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: Bootstrap starting.
I20260925 02:48:52.024768 24846 tablet_bootstrap.cc:492] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: Bootstrap starting.
I20260925 02:48:52.026525 24847 tablet_bootstrap.cc:492] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: Bootstrap starting.
I20260925 02:48:52.027406 24845 tablet_bootstrap.cc:654] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:52.028434 24846 tablet_bootstrap.cc:654] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:52.028761 24845 log.cc:824] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:52.029692 24846 log.cc:824] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:52.030066 24847 tablet_bootstrap.cc:654] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:52.031117 24847 log.cc:824] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:52.032419 24845 tablet_bootstrap.cc:492] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: No bootstrap required, opened a new log
I20260925 02:48:52.032708 24845 ts_tablet_manager.cc:1397] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260925 02:48:52.033818 24846 tablet_bootstrap.cc:492] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: No bootstrap required, opened a new log
I20260925 02:48:52.034082 24846 ts_tablet_manager.cc:1397] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260925 02:48:52.034265 24847 tablet_bootstrap.cc:492] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: No bootstrap required, opened a new log
I20260925 02:48:52.034557 24847 ts_tablet_manager.cc:1397] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260925 02:48:52.040783 24845 raft_consensus.cc:348] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90795bb485344ebba300a00c516ac67a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44559 } } peers { permanent_uuid: "501f57d1365c47909e7fa077a141961f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41623 } } peers { permanent_uuid: "8a0ff573c0c34998ac077c553920f8d4" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38443 } }
I20260925 02:48:52.041203 24845 raft_consensus.cc:374] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:52.041316 24845 raft_consensus.cc:729] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90795bb485344ebba300a00c516ac67a, State: Initialized, Role: FOLLOWER
I20260925 02:48:52.041971 24845 consensus_queue.cc:260] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90795bb485344ebba300a00c516ac67a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44559 } } peers { permanent_uuid: "501f57d1365c47909e7fa077a141961f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41623 } } peers { permanent_uuid: "8a0ff573c0c34998ac077c553920f8d4" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38443 } }
I20260925 02:48:52.043933 24845 ts_tablet_manager.cc:1428] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260925 02:48:52.044864 24846 raft_consensus.cc:348] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90795bb485344ebba300a00c516ac67a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44559 } } peers { permanent_uuid: "501f57d1365c47909e7fa077a141961f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41623 } } peers { permanent_uuid: "8a0ff573c0c34998ac077c553920f8d4" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38443 } }
I20260925 02:48:52.045015 24847 raft_consensus.cc:348] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90795bb485344ebba300a00c516ac67a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44559 } } peers { permanent_uuid: "501f57d1365c47909e7fa077a141961f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41623 } } peers { permanent_uuid: "8a0ff573c0c34998ac077c553920f8d4" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38443 } }
I20260925 02:48:52.045333 24846 raft_consensus.cc:374] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:52.045457 24847 raft_consensus.cc:374] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:52.045445 24846 raft_consensus.cc:729] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 501f57d1365c47909e7fa077a141961f, State: Initialized, Role: FOLLOWER
I20260925 02:48:52.045557 24847 raft_consensus.cc:729] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a0ff573c0c34998ac077c553920f8d4, State: Initialized, Role: FOLLOWER
I20260925 02:48:52.046123 24846 consensus_queue.cc:260] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90795bb485344ebba300a00c516ac67a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44559 } } peers { permanent_uuid: "501f57d1365c47909e7fa077a141961f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41623 } } peers { permanent_uuid: "8a0ff573c0c34998ac077c553920f8d4" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38443 } }
I20260925 02:48:52.046118 24847 consensus_queue.cc:260] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90795bb485344ebba300a00c516ac67a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44559 } } peers { permanent_uuid: "501f57d1365c47909e7fa077a141961f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41623 } } peers { permanent_uuid: "8a0ff573c0c34998ac077c553920f8d4" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38443 } }
I20260925 02:48:52.047825 24826 heartbeater.cc:503] Master 127.5.56.190:34469 was elected leader, sending a full tablet report...
I20260925 02:48:52.048056 24846 ts_tablet_manager.cc:1428] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260925 02:48:52.048285 24847 ts_tablet_manager.cc:1428] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260925 02:48:52.048688 24695 heartbeater.cc:503] Master 127.5.56.190:34469 was elected leader, sending a full tablet report...
W20260925 02:48:52.170908 24827 tablet.cc:2369] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:48:52.200802 24696 tablet.cc:2369] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 02:48:52.222527 24564 tablet.cc:2369] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:52.256193 24852 raft_consensus.cc:482] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:52.256558 24852 raft_consensus.cc:504] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90795bb485344ebba300a00c516ac67a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44559 } } peers { permanent_uuid: "501f57d1365c47909e7fa077a141961f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41623 } } peers { permanent_uuid: "8a0ff573c0c34998ac077c553920f8d4" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38443 } }
I20260925 02:48:52.258271 24852 leader_election.cc:288] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90795bb485344ebba300a00c516ac67a (127.5.56.129:44559), 8a0ff573c0c34998ac077c553920f8d4 (127.5.56.131:38443)
I20260925 02:48:52.265194 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:52.258799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:54417 (local address 127.5.56.129:44559)
0925 02:48:52.259005 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:52.259009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:52.259675 (+ 666us) server_negotiation.cc:407] Connection header received
0925 02:48:52.259971 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:52.259980 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:52.260091 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:52.260158 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:52.260913 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.261594 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.263481 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.264250 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.264443 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.264521 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:52.264804 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:48:52.264941 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:48:52.265328 24854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:52.258784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:44559 (local address 127.5.56.130:54417)
0925 02:48:52.259552 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:52.259574 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:52.259735 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:52.260361 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:52.260364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:52.260387 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:52.260746 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:52.260754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.261735 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.261742 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:52.263270 (+ 1528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:52.263285 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.264447 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.264453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:52.264630 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.264654 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:48:52.264761 (+ 107us) client_negotiation.cc:240] Negotiation successful
0925 02:48:52.264901 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":620,"spinlock_wait_cycles":33408,"thread_start_us":395,"threads_started":1}
I20260925 02:48:52.265553 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:52.260044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:43929 (local address 127.5.56.131:38443)
0925 02:48:52.260267 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:52.260274 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:52.260475 (+ 201us) server_negotiation.cc:407] Connection header received
0925 02:48:52.260711 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:52.260716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:52.260785 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:52.260903 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:52.261417 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.262133 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.263891 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.264674 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.264866 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.264933 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:52.265204 (+ 271us) server_negotiation.cc:299] Negotiation successful
0925 02:48:52.265328 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 02:48:52.265784 24855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:52.259717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:38443 (local address 127.5.56.130:43929)
0925 02:48:52.260363 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:52.260377 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:52.260567 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:52.260918 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:52.260922 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:52.260944 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:52.261266 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:52.261275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.262278 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.262288 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:52.263707 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:52.263718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.264865 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.264872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:52.265044 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.265068 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:48:52.265173 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 02:48:52.265312 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":304,"threads_started":1}
I20260925 02:48:52.266548 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba26d389d594230bff5a285090d81fa" candidate_uuid: "501f57d1365c47909e7fa077a141961f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90795bb485344ebba300a00c516ac67a" is_pre_election: true
I20260925 02:48:52.266808 24781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba26d389d594230bff5a285090d81fa" candidate_uuid: "501f57d1365c47909e7fa077a141961f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a0ff573c0c34998ac077c553920f8d4" is_pre_election: true
I20260925 02:48:52.267064 24516 raft_consensus.cc:2454] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 501f57d1365c47909e7fa077a141961f in term 0.
I20260925 02:48:52.267359 24781 raft_consensus.cc:2454] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 501f57d1365c47909e7fa077a141961f in term 0.
I20260925 02:48:52.267840 24584 leader_election.cc:302] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501f57d1365c47909e7fa077a141961f, 90795bb485344ebba300a00c516ac67a; no voters:
I20260925 02:48:52.268210 24852 raft_consensus.cc:2781] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:52.268369 24852 raft_consensus.cc:482] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:52.268474 24852 raft_consensus.cc:3037] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:52.270805 24852 raft_consensus.cc:504] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90795bb485344ebba300a00c516ac67a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44559 } } peers { permanent_uuid: "501f57d1365c47909e7fa077a141961f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41623 } } peers { permanent_uuid: "8a0ff573c0c34998ac077c553920f8d4" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38443 } }
I20260925 02:48:52.271418 24852 leader_election.cc:288] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [CANDIDATE]: Term 1 election: Requested vote from peers 90795bb485344ebba300a00c516ac67a (127.5.56.129:44559), 8a0ff573c0c34998ac077c553920f8d4 (127.5.56.131:38443)
I20260925 02:48:52.272042 24781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba26d389d594230bff5a285090d81fa" candidate_uuid: "501f57d1365c47909e7fa077a141961f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a0ff573c0c34998ac077c553920f8d4"
I20260925 02:48:52.272037 24516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cba26d389d594230bff5a285090d81fa" candidate_uuid: "501f57d1365c47909e7fa077a141961f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90795bb485344ebba300a00c516ac67a"
I20260925 02:48:52.272266 24781 raft_consensus.cc:3037] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:52.272293 24516 raft_consensus.cc:3037] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:52.275624 24516 raft_consensus.cc:2454] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501f57d1365c47909e7fa077a141961f in term 1.
I20260925 02:48:52.275624 24781 raft_consensus.cc:2454] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 501f57d1365c47909e7fa077a141961f in term 1.
I20260925 02:48:52.276096 24584 leader_election.cc:302] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 501f57d1365c47909e7fa077a141961f, 8a0ff573c0c34998ac077c553920f8d4; no voters:
I20260925 02:48:52.276490 24852 raft_consensus.cc:2781] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:52.277009 24852 raft_consensus.cc:686] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 1 LEADER]: Becoming Leader. State: Replica: 501f57d1365c47909e7fa077a141961f, State: Running, Role: LEADER
I20260925 02:48:52.277504 24852 consensus_queue.cc:237] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90795bb485344ebba300a00c516ac67a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44559 } } peers { permanent_uuid: "501f57d1365c47909e7fa077a141961f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41623 } } peers { permanent_uuid: "8a0ff573c0c34998ac077c553920f8d4" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38443 } }
I20260925 02:48:52.282783 24375 catalog_manager.cc:5495] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f reported cstate change: term changed from 0 to 1, leader changed from <none> to 501f57d1365c47909e7fa077a141961f (127.5.56.130). New cstate: current_term: 1 leader_uuid: "501f57d1365c47909e7fa077a141961f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90795bb485344ebba300a00c516ac67a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "501f57d1365c47909e7fa077a141961f" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 41623 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8a0ff573c0c34998ac077c553920f8d4" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38443 } health_report { overall_health: UNKNOWN } } }
I20260925 02:48:52.366549 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:52.361131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57700 (local address 127.5.56.130:41623)
0925 02:48:52.361323 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:52.361328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:52.361347 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:48:52.361430 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:52.361434 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:52.361521 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:52.361648 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:52.361989 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.362778 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.364114 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.364538 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.364646 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.365216 (+ 570us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:52.365835 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:52.365982 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:52.366156 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 02:48:52.366208 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260925 02:48:52.375114 24781 raft_consensus.cc:1261] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4 [term 1 FOLLOWER]: Refusing update from remote peer 501f57d1365c47909e7fa077a141961f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:52.375114 24516 raft_consensus.cc:1261] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a [term 1 FOLLOWER]: Refusing update from remote peer 501f57d1365c47909e7fa077a141961f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:52.376235 24852 consensus_queue.cc:1035] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [LEADER]: Connected to new peer: Peer: permanent_uuid: "90795bb485344ebba300a00c516ac67a" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 44559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:52.376564 24856 consensus_queue.cc:1035] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a0ff573c0c34998ac077c553920f8d4" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 38443 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:52.405550 24374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:52.405892 24374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49870:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:52.409955 24374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a0ba076d56f14b878fc1dc74ce245ef5: Sending DeleteTablet for 3 replicas of tablet cba26d389d594230bff5a285090d81fa
I20260925 02:48:52.411271 24630 tablet_service.cc:1514] Processing DeleteTablet for tablet cba26d389d594230bff5a285090d81fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:52 UTC) from {username='slave'} at 127.0.0.1:57690
I20260925 02:48:52.413352 24496 tablet_service.cc:1514] Processing DeleteTablet for tablet cba26d389d594230bff5a285090d81fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:52 UTC) from {username='slave'} at 127.0.0.1:39652
I20260925 02:48:52.413609 24869 tablet_replica.cc:330] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: stopping tablet replica
I20260925 02:48:52.414109 24869 raft_consensus.cc:2229] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:52.414601 24761 tablet_service.cc:1514] Processing DeleteTablet for tablet cba26d389d594230bff5a285090d81fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:52 UTC) from {username='slave'} at 127.0.0.1:36702
I20260925 02:48:52.414860 24869 raft_consensus.cc:2258] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:52.417217 24869 ts_tablet_manager.cc:1905] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:52.417910 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 24434
I20260925 02:48:52.426802 24869 ts_tablet_manager.cc:1918] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:52.427592 24870 tablet_replica.cc:330] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: stopping tablet replica
I20260925 02:48:52.428153 24870 raft_consensus.cc:2229] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:52.428757 24870 raft_consensus.cc:2258] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:52.431135 24870 ts_tablet_manager.cc:1905] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:52.438406 24870 ts_tablet_manager.cc:1918] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:52.438742 24869 log.cc:1196] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/cba26d389d594230bff5a285090d81fa
I20260925 02:48:52.439410 24869 ts_tablet_manager.cc:1939] T cba26d389d594230bff5a285090d81fa P 501f57d1365c47909e7fa077a141961f: Deleting consensus metadata
I20260925 02:48:52.440699 24870 log.cc:1196] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/cba26d389d594230bff5a285090d81fa
I20260925 02:48:52.441386 24870 ts_tablet_manager.cc:1939] T cba26d389d594230bff5a285090d81fa P 90795bb485344ebba300a00c516ac67a: Deleting consensus metadata
I20260925 02:48:52.441557 24359 catalog_manager.cc:4841] TS 501f57d1365c47909e7fa077a141961f (127.5.56.130:41623): tablet cba26d389d594230bff5a285090d81fa (table test-table [id=96abb60105c54c56994bda84bd37b007]) successfully deleted
I20260925 02:48:52.443383 24361 catalog_manager.cc:4841] TS 90795bb485344ebba300a00c516ac67a (127.5.56.129:44559): tablet cba26d389d594230bff5a285090d81fa (table test-table [id=96abb60105c54c56994bda84bd37b007]) successfully deleted
I20260925 02:48:52.443667 24871 tablet_replica.cc:330] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: stopping tablet replica
I20260925 02:48:52.444137 24871 raft_consensus.cc:2229] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:52.444629 24871 raft_consensus.cc:2258] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:52.446557 24871 ts_tablet_manager.cc:1905] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:52.454432 24871 ts_tablet_manager.cc:1918] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:52.454610 24871 log.cc:1196] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/cba26d389d594230bff5a285090d81fa
I20260925 02:48:52.455071 24871 ts_tablet_manager.cc:1939] T cba26d389d594230bff5a285090d81fa P 8a0ff573c0c34998ac077c553920f8d4: Deleting consensus metadata
I20260925 02:48:52.456977 24361 catalog_manager.cc:4841] TS 8a0ff573c0c34998ac077c553920f8d4 (127.5.56.131:38443): tablet cba26d389d594230bff5a285090d81fa (table test-table [id=96abb60105c54c56994bda84bd37b007]) successfully deleted
I20260925 02:48:52.459095 24840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:52.419374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39668 (local address 127.5.56.129:44559)
0925 02:48:52.419606 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:52.419615 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:52.442521 (+ 22906us) server_negotiation.cc:407] Connection header received
0925 02:48:52.442851 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:52.442856 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:52.442964 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:52.443047 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:52.443931 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.445059 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.447196 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.447842 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.447938 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.457200 (+ 9262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:52.457215 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:52.457235 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:52.458049 (+ 814us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:52.458121 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:52.458162 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:52.458270 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:52.458605 (+ 335us) server_negotiation.cc:299] Negotiation successful
0925 02:48:52.458740 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:48:52.459955 24558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:52.555918 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 24434
I20260925 02:48:52.580853 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 24567
I20260925 02:48:52.588541 24841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:52.581675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.5.56.130:41623)
0925 02:48:52.581854 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:52.581861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:52.581960 (+ 99us) server_negotiation.cc:407] Connection header received
0925 02:48:52.582179 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:52.582185 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:52.582281 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:52.582427 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:52.582847 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.583620 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.585051 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.585604 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.585768 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.587032 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:52.587041 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:52.587055 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:52.587708 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:52.587758 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:52.587778 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:52.587901 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:52.588044 (+ 143us) server_negotiation.cc:299] Negotiation successful
0925 02:48:52.588157 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:48:52.589298 24690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:52.704860 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 24567
I20260925 02:48:52.726150 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 24698
I20260925 02:48:52.732920 24844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:52.726933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36708 (local address 127.5.56.131:38443)
0925 02:48:52.727113 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:52.727118 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:52.727137 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:48:52.727297 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:52.727301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:52.727395 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:52.727489 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:52.727896 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.728633 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.729957 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:52.730370 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:52.730669 (+ 299us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:52.731652 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:52.731661 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:52.731679 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:52.732268 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:52.732321 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:52.732342 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:52.732429 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:52.732539 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 02:48:52.732630 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:48:52.733569 24821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:52.824043 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 24698
I20260925 02:48:52.842567 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 24343
I20260925 02:48:52.843675 24405 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:52.955440 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 24343
2026-09-25T02:48:52Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3070 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:48:53 dist-test-slave-n563 krb5kdc[24879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:48:53 dist-test-slave-n563 krb5kdc[24879](info): set up 2 sockets
Sep 25 02:48:53 dist-test-slave-n563 krb5kdc[24879](info): commencing operation
krb5kdc: starting...
W20260925 02:48:54.083900 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.085s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:48:54 dist-test-slave-n563 krb5kdc[24879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:48:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:48:54.360448 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43037
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:43037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:54.709270 24892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:54.709604 24892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:54.709772 24892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:54.717221 24892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:54.717346 24892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:54.717455 24892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:54.717562 24892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:54.717643 24892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:54.728127 24892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:43037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:54.729594 24892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:54.731010 24892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:54.743561 24897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:54.743505 24900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:54.743539 24898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:54.744653 24892 server_base.cc:1034] running on GCE node
I20260925 02:48:54.745582 24892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:54.747267 24892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:54.748518 24892 hybrid_clock.cc:648] HybridClock initialized: now 1790304534748474 us; error 56 us; skew 500 ppm
I20260925 02:48:54.748929 24892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:54 dist-test-slave-n563 krb5kdc[24879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304534, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:54.752192 24892 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:48:54.753767 24892 webserver.cc:460] Webserver started at http://127.5.56.190:33671/ using document root <none> and password file <none>
I20260925 02:48:54.754558 24892 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:54.754712 24892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:54.755048 24892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:54.758284 24892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "e092db2a0fe841188d8e9ee0f3228979"
format_stamp: "Formatted at 2026-09-25 02:48:54 on dist-test-slave-n563"
server_key: "030d2070957ec9da2dbef2b9fd22fb92"
server_key_iv: "ecc52f8923b582e3efc8c647a1557ce9"
server_key_version: "encryptionkey@0"
I20260925 02:48:54.759256 24892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "e092db2a0fe841188d8e9ee0f3228979"
format_stamp: "Formatted at 2026-09-25 02:48:54 on dist-test-slave-n563"
server_key: "030d2070957ec9da2dbef2b9fd22fb92"
server_key_iv: "ecc52f8923b582e3efc8c647a1557ce9"
server_key_version: "encryptionkey@0"
I20260925 02:48:54.764822 24892 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:48:54.768666 24907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:54.770107 24892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:48:54.770311 24892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "e092db2a0fe841188d8e9ee0f3228979"
format_stamp: "Formatted at 2026-09-25 02:48:54 on dist-test-slave-n563"
server_key: "030d2070957ec9da2dbef2b9fd22fb92"
server_key_iv: "ecc52f8923b582e3efc8c647a1557ce9"
server_key_version: "encryptionkey@0"
I20260925 02:48:54.770586 24892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:54.781910 24892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:54.786036 24892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:54.786358 24892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:54.805713 24892 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:43037
I20260925 02:48:54.805737 24958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:43037 every 8 connection(s)
I20260925 02:48:54.807350 24892 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:54.808058 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 24892
I20260925 02:48:54.808234 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:48:54.808838 5346 external_mini_cluster.cc:1428] Setting key 29270a5abf54e3f00794d893d708d1b8
I20260925 02:48:54.812119 24959 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:48:54 dist-test-slave-n563 krb5kdc[24879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:54.823091 24959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979: Bootstrap starting.
I20260925 02:48:54.826758 24959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:54.827873 24959 log.cc:824] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:54.831041 24959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979: No bootstrap required, opened a new log
I20260925 02:48:54.831629 24962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:54.810839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48036 (local address 127.5.56.190:43037)
0925 02:48:54.811639 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:54.811652 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:54.811694 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:48:54.813500 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:54.813524 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:54.813957 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:54.814371 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:54.815934 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:54.817329 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:54.818718 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:54.819410 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:54.819616 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:54.823892 (+ 4276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:54.823901 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:54.823917 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:54.823955 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:54.827278 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:54.828080 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:54.828088 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:54.828094 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:54.828266 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:54.828694 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:54.828697 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:54.828699 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:54.829254 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:54.829597 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:54.830042 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 02:48:54.830530 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":362,"threads_started":1}
I20260925 02:48:54.839424 24959 raft_consensus.cc:348] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e092db2a0fe841188d8e9ee0f3228979" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43037 } }
I20260925 02:48:54.839911 24959 raft_consensus.cc:374] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:54.840032 24959 raft_consensus.cc:729] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e092db2a0fe841188d8e9ee0f3228979, State: Initialized, Role: FOLLOWER
I20260925 02:48:54.840771 24959 consensus_queue.cc:260] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e092db2a0fe841188d8e9ee0f3228979" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43037 } }
I20260925 02:48:54.841074 24959 raft_consensus.cc:388] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:54.841251 24959 raft_consensus.cc:482] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:54.841450 24959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:54.843782 24959 raft_consensus.cc:504] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e092db2a0fe841188d8e9ee0f3228979" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43037 } }
I20260925 02:48:54.844262 24959 leader_election.cc:302] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e092db2a0fe841188d8e9ee0f3228979; no voters:
I20260925 02:48:54.844872 24959 leader_election.cc:288] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:54.845024 24964 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:54.845731 24964 raft_consensus.cc:686] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [term 1 LEADER]: Becoming Leader. State: Replica: e092db2a0fe841188d8e9ee0f3228979, State: Running, Role: LEADER
I20260925 02:48:54.846413 24964 consensus_queue.cc:237] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e092db2a0fe841188d8e9ee0f3228979" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43037 } }
I20260925 02:48:54.847314 24959 sys_catalog.cc:564] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:54.849709 24966 sys_catalog.cc:455] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e092db2a0fe841188d8e9ee0f3228979. Latest consensus state: current_term: 1 leader_uuid: "e092db2a0fe841188d8e9ee0f3228979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e092db2a0fe841188d8e9ee0f3228979" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43037 } } }
I20260925 02:48:54.850221 24966 sys_catalog.cc:458] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:54.850884 24965 sys_catalog.cc:455] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e092db2a0fe841188d8e9ee0f3228979" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e092db2a0fe841188d8e9ee0f3228979" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43037 } } }
I20260925 02:48:54.851297 24965 sys_catalog.cc:458] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979 [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:54.855787 24971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:54.861635 24971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:54.870893 24971 catalog_manager.cc:1347] Generated new cluster ID: 2dba9c7b499f45baaf706038ba02386f
I20260925 02:48:54.871023 24971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:54.885392 24971 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:54.886893 24971 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:54.901005 24971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979: Generated new TSK 0
I20260925 02:48:54.902120 24971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.5.56.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
Entry for principal kudu/127.5.56.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab.
I20260925 02:48:54.985975 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43037
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:55.347424 24986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:55.347765 24986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:55.347941 24986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:55.355901 24986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:55.356055 24986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:55.356238 24986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:48:55.366840 24986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.129.keytab
--principal=kudu/127.5.56.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:55.368250 24986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:55.369618 24986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:55.383158 24991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:55.383076 24992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:55.383113 24994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:55.383948 24986 server_base.cc:1034] running on GCE node
I20260925 02:48:55.384537 24986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:55.385524 24986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:55.386724 24986 hybrid_clock.cc:648] HybridClock initialized: now 1790304535386674 us; error 67 us; skew 500 ppm
I20260925 02:48:55.387084 24986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:55 dist-test-slave-n563 krb5kdc[24879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304535, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:55.390348 24986 init.cc:377] Logged in from keytab as kudu/127.5.56.129@KRBTEST.COM (short username kudu)
I20260925 02:48:55.391913 24986 webserver.cc:460] Webserver started at http://127.5.56.129:33301/ using document root <none> and password file <none>
I20260925 02:48:55.392625 24986 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:55.392781 24986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:55.393100 24986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:55.396272 24986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "130a2ae994614e028d717bdb0f53b1d4"
format_stamp: "Formatted at 2026-09-25 02:48:55 on dist-test-slave-n563"
server_key: "1492ca526fa6a5b71fbd371fd8765473"
server_key_iv: "134e9ffb1e52c01423a61ba1f97ad792"
server_key_version: "encryptionkey@0"
I20260925 02:48:55.397148 24986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "130a2ae994614e028d717bdb0f53b1d4"
format_stamp: "Formatted at 2026-09-25 02:48:55 on dist-test-slave-n563"
server_key: "1492ca526fa6a5b71fbd371fd8765473"
server_key_iv: "134e9ffb1e52c01423a61ba1f97ad792"
server_key_version: "encryptionkey@0"
I20260925 02:48:55.402707 24986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260925 02:48:55.406282 25001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:55.407783 24986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:55.407991 24986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "130a2ae994614e028d717bdb0f53b1d4"
format_stamp: "Formatted at 2026-09-25 02:48:55 on dist-test-slave-n563"
server_key: "1492ca526fa6a5b71fbd371fd8765473"
server_key_iv: "134e9ffb1e52c01423a61ba1f97ad792"
server_key_version: "encryptionkey@0"
I20260925 02:48:55.408216 24986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:55.444406 24986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:55.449375 24986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:55.449692 24986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:55.450788 24986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:55.452179 24986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:55.452283 24986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:55.452431 24986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:55.452504 24986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:55.486909 24986 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:37753
I20260925 02:48:55.486949 25113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:37753 every 8 connection(s)
I20260925 02:48:55.488344 24986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:55.498081 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 24986
I20260925 02:48:55.498291 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:55.498835 5346 external_mini_cluster.cc:1428] Setting key 3eb8e078458c8f9d35971d35f25c7e59
Sep 25 02:48:55 dist-test-slave-n563 krb5kdc[24879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304535, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.129@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:55.511860 25116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:55.491067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:60793 (local address 127.5.56.190:43037)
0925 02:48:55.491529 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:55.491534 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:55.492447 (+ 913us) server_negotiation.cc:407] Connection header received
0925 02:48:55.494556 (+ 2109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:55.494563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:55.494683 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:55.494819 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:55.496515 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:55.497320 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:55.499506 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:55.500185 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:55.500325 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:55.504636 (+ 4311us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:55.504645 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:55.504652 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:55.504699 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:55.508383 (+ 3684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:55.509266 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:55.509274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:55.509281 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:55.509475 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:55.509921 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:55.509928 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:55.509934 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:55.510344 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:55.510527 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:55.511323 (+ 796us) server_negotiation.cc:299] Negotiation successful
0925 02:48:55.511519 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":249,"threads_started":1}
I20260925 02:48:55.513695 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:55.491446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43037 (local address 127.5.56.129:60793)
0925 02:48:55.492240 (+ 794us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:55.492287 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:55.494192 (+ 1905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:55.495085 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:55.495101 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:55.495674 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:55.496309 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:55.496329 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:55.497503 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:55.497508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:55.499291 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:55.499311 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:55.500443 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:55.500448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:55.500663 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:55.501498 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:55.501529 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:55.504371 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:55.508584 (+ 4213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:55.508592 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:55.508608 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:55.509098 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:55.509589 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:55.509593 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:55.509595 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:55.509795 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:55.510500 (+ 705us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:55.510507 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:55.510881 (+ 374us) client_negotiation.cc:769] Sending connection context
0925 02:48:55.511199 (+ 318us) client_negotiation.cc:240] Negotiation successful
0925 02:48:55.511892 (+ 693us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":314,"threads_started":1}
I20260925 02:48:55.515450 25114 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43037
I20260925 02:48:55.515789 25114 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:55.516544 25114 heartbeater.cc:511] Master 127.5.56.190:43037 requested a full tablet report, sending...
I20260925 02:48:55.518788 24924 ts_manager.cc:194] Registered new tserver with Master: 130a2ae994614e028d717bdb0f53b1d4 (127.5.56.129:37753)
I20260925 02:48:55.520274 24924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.129@KRBTEST.COM'} at 127.5.56.129:60793
WARNING: no policy specified for kudu/127.5.56.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
Entry for principal kudu/127.5.56.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab.
I20260925 02:48:55.563158 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43037
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:55.931334 25121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:55.931649 25121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:55.931746 25121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:55.939157 25121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:55.939278 25121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:55.939393 25121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:48:55.949684 25121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.130.keytab
--principal=kudu/127.5.56.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:55.951013 25121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:55.952306 25121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:55.966413 25126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:55.966364 25129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:55.966347 25127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:55.967128 25121 server_base.cc:1034] running on GCE node
I20260925 02:48:55.967993 25121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:55.968955 25121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:55.970225 25121 hybrid_clock.cc:648] HybridClock initialized: now 1790304535970162 us; error 57 us; skew 500 ppm
I20260925 02:48:55.970626 25121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:55 dist-test-slave-n563 krb5kdc[24879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304535, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:55.973831 25121 init.cc:377] Logged in from keytab as kudu/127.5.56.130@KRBTEST.COM (short username kudu)
I20260925 02:48:55.975628 25121 webserver.cc:460] Webserver started at http://127.5.56.130:46441/ using document root <none> and password file <none>
I20260925 02:48:55.976364 25121 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:55.976516 25121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:55.976848 25121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:55.980134 25121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "da3de2ea6d804451a3f93a0bee35e4d2"
format_stamp: "Formatted at 2026-09-25 02:48:55 on dist-test-slave-n563"
server_key: "575fe9e73d98be427d46be0dd59b27b4"
server_key_iv: "5f4ba2de9a9791f2e311681b9e4020ad"
server_key_version: "encryptionkey@0"
I20260925 02:48:55.980947 25121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "da3de2ea6d804451a3f93a0bee35e4d2"
format_stamp: "Formatted at 2026-09-25 02:48:55 on dist-test-slave-n563"
server_key: "575fe9e73d98be427d46be0dd59b27b4"
server_key_iv: "5f4ba2de9a9791f2e311681b9e4020ad"
server_key_version: "encryptionkey@0"
I20260925 02:48:55.986465 25121 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260925 02:48:55.990043 25136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:55.991489 25121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 02:48:55.991675 25121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "da3de2ea6d804451a3f93a0bee35e4d2"
format_stamp: "Formatted at 2026-09-25 02:48:55 on dist-test-slave-n563"
server_key: "575fe9e73d98be427d46be0dd59b27b4"
server_key_iv: "5f4ba2de9a9791f2e311681b9e4020ad"
server_key_version: "encryptionkey@0"
I20260925 02:48:55.991902 25121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:56.017537 25121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:56.021786 25121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:56.022097 25121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:56.023176 25121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:56.024526 25121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:56.024636 25121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:56.024791 25121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:56.024864 25121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:56.058916 25121 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:38155
I20260925 02:48:56.058971 25248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:38155 every 8 connection(s)
I20260925 02:48:56.060263 25121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:56.060958 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 25121
I20260925 02:48:56.061156 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:56.061888 5346 external_mini_cluster.cc:1428] Setting key 7d75c3cd17b29468576c9427ffb10d9e
Sep 25 02:48:56 dist-test-slave-n563 krb5kdc[24879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304535, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.130@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:56.085731 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.063163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:35377 (local address 127.5.56.190:43037)
0925 02:48:56.063639 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:56.063644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:56.064904 (+ 1260us) server_negotiation.cc:407] Connection header received
0925 02:48:56.067901 (+ 2997us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:56.067906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:56.067972 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:56.068065 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:56.070221 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.071032 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.073667 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.074146 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.074245 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.078368 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:56.078377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:56.078402 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:56.078449 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:56.082158 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:56.083074 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:56.083082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:56.083088 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:56.083250 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:56.083679 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:56.083687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:56.083693 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:56.084105 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:56.084268 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:56.084977 (+ 709us) server_negotiation.cc:299] Negotiation successful
0925 02:48:56.085245 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":255,"threads_started":1}
I20260925 02:48:56.086515 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.063776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43037 (local address 127.5.56.130:35377)
0925 02:48:56.064709 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:56.064766 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:56.067597 (+ 2831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:56.068347 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:56.068362 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:56.069153 (+ 791us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:56.070028 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.070057 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.071186 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.071195 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.073386 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.073405 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.074405 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.074415 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.074764 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.075633 (+ 869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:56.075658 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:56.078098 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:56.082333 (+ 4235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:56.082341 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:56.082358 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:56.082933 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:56.083365 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:56.083368 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:56.083370 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:56.083557 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:56.084238 (+ 681us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:56.084244 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:56.084582 (+ 338us) client_negotiation.cc:769] Sending connection context
0925 02:48:56.084816 (+ 234us) client_negotiation.cc:240] Negotiation successful
0925 02:48:56.085251 (+ 435us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":371,"threads_started":1}
I20260925 02:48:56.088217 25249 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43037
I20260925 02:48:56.088577 25249 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:56.089478 25249 heartbeater.cc:511] Master 127.5.56.190:43037 requested a full tablet report, sending...
I20260925 02:48:56.091176 24924 ts_manager.cc:194] Registered new tserver with Master: da3de2ea6d804451a3f93a0bee35e4d2 (127.5.56.130:38155)
I20260925 02:48:56.092025 24924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.130@KRBTEST.COM'} at 127.5.56.130:35377
WARNING: no policy specified for kudu/127.5.56.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
Entry for principal kudu/127.5.56.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab.
I20260925 02:48:56.128068 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43037
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:48:56.472517 25255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:56.472892 25255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:56.473069 25255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:56.480872 25255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:56.481060 25255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:56.481248 25255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:48:56.492005 25255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.131.keytab
--principal=kudu/127.5.56.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:43037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:56.493342 25255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:56.494747 25255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:56.508730 25260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:56.508834 25261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:56.508785 25263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:56.509697 25255 server_base.cc:1034] running on GCE node
I20260925 02:48:56.510311 25255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:56.511343 25255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:56.512539 25255 hybrid_clock.cc:648] HybridClock initialized: now 1790304536512490 us; error 44 us; skew 500 ppm
I20260925 02:48:56.512835 25255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:48:56 dist-test-slave-n563 krb5kdc[24879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304536, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:48:56.516126 25255 init.cc:377] Logged in from keytab as kudu/127.5.56.131@KRBTEST.COM (short username kudu)
I20260925 02:48:56.517663 25255 webserver.cc:460] Webserver started at http://127.5.56.131:38083/ using document root <none> and password file <none>
I20260925 02:48:56.518352 25255 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:56.518527 25255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:56.518819 25255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:56.521798 25255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "7d7d24fe4aa9455286b162b235126bb1"
format_stamp: "Formatted at 2026-09-25 02:48:56 on dist-test-slave-n563"
server_key: "ca0ad271d5e794796b46eef5697605a8"
server_key_iv: "65c0cd0a523da54e6a92ad15067560f2"
server_key_version: "encryptionkey@0"
I20260925 02:48:56.522645 25255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "7d7d24fe4aa9455286b162b235126bb1"
format_stamp: "Formatted at 2026-09-25 02:48:56 on dist-test-slave-n563"
server_key: "ca0ad271d5e794796b46eef5697605a8"
server_key_iv: "65c0cd0a523da54e6a92ad15067560f2"
server_key_version: "encryptionkey@0"
I20260925 02:48:56.523478 25114 heartbeater.cc:503] Master 127.5.56.190:43037 was elected leader, sending a full tablet report...
I20260925 02:48:56.528566 25255 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260925 02:48:56.532159 25270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:56.533558 25255 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260925 02:48:56.533701 25255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "7d7d24fe4aa9455286b162b235126bb1"
format_stamp: "Formatted at 2026-09-25 02:48:56 on dist-test-slave-n563"
server_key: "ca0ad271d5e794796b46eef5697605a8"
server_key_iv: "65c0cd0a523da54e6a92ad15067560f2"
server_key_version: "encryptionkey@0"
I20260925 02:48:56.533891 25255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:56.552721 25255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:56.557186 25255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:56.557439 25255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:56.558466 25255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:56.559852 25255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:56.559924 25255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:56.560019 25255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:56.560065 25255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:56.595052 25255 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:32777
I20260925 02:48:56.595165 25384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:32777 every 8 connection(s)
I20260925 02:48:56.596581 25255 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:56.605014 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 25255
I20260925 02:48:56.605227 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:56.605726 5346 external_mini_cluster.cc:1428] Setting key e020f85bffcdbe53416cc4df435c2f82
Sep 25 02:48:56 dist-test-slave-n563 krb5kdc[24879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304536, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.131@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:48:56.617389 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.599335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:34133 (local address 127.5.56.190:43037)
0925 02:48:56.599773 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:56.599778 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:56.600794 (+ 1016us) server_negotiation.cc:407] Connection header received
0925 02:48:56.602754 (+ 1960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:56.602760 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:56.602880 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:56.603003 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:56.604577 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.605464 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.607336 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.607778 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.607915 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.611390 (+ 3475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:56.611397 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:56.611402 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:56.611470 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:56.614197 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:56.615012 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:56.615020 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:56.615029 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:56.615149 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:56.615592 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:56.615599 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:56.615606 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:56.615887 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:56.616052 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:56.616750 (+ 698us) server_negotiation.cc:299] Negotiation successful
0925 02:48:56.616983 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":216,"threads_started":1}
I20260925 02:48:56.618374 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.599902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:43037 (local address 127.5.56.131:34133)
0925 02:48:56.600600 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:56.600645 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:56.602475 (+ 1830us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:56.603274 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:56.603282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:56.603808 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:56.604398 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.604416 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.605625 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.605629 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.607199 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.607212 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.608009 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.608017 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.608259 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.608906 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:48:56.608934 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:56.611164 (+ 2230us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:56.614353 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:56.614362 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:56.614428 (+ 66us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:56.614877 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:56.615262 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:48:56.615267 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:48:56.615271 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:48:56.615469 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:48:56.616017 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:56.616026 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:56.616392 (+ 366us) client_negotiation.cc:769] Sending connection context
0925 02:48:56.616661 (+ 269us) client_negotiation.cc:240] Negotiation successful
0925 02:48:56.617224 (+ 563us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":272,"threads_started":1}
I20260925 02:48:56.619894 25385 heartbeater.cc:347] Connected to a master server at 127.5.56.190:43037
I20260925 02:48:56.620210 25385 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:56.620944 25385 heartbeater.cc:511] Master 127.5.56.190:43037 requested a full tablet report, sending...
I20260925 02:48:56.622344 24924 ts_manager.cc:194] Registered new tserver with Master: 7d7d24fe4aa9455286b162b235126bb1 (127.5.56.131:32777)
I20260925 02:48:56.623214 24924 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.5.56.131@KRBTEST.COM'} at 127.5.56.131:34133
I20260925 02:48:56.633647 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:48:56.650708 25387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.638129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48050 (local address 127.5.56.190:43037)
0925 02:48:56.638403 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:56.638413 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:56.638896 (+ 483us) server_negotiation.cc:407] Connection header received
0925 02:48:56.639038 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:56.639042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:56.639130 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:56.639217 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:56.640666 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.641393 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.642747 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.643161 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.643266 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.645516 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:56.645521 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:56.645524 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:56.645555 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:56.648259 (+ 2704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:56.648936 (+ 677us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:56.648940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:56.648942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:56.649060 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:56.649378 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:56.649381 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:56.649383 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:56.649703 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:56.649898 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:56.650135 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 02:48:56.650304 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 02:48:56.654675 24924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:56.661868 24924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:56.695911 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.675047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:38155 (local address 127.0.0.1:47342)
0925 02:48:56.686854 (+ 11807us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:56.686874 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:56.687027 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:56.688138 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:56.688146 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:56.688179 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:56.689015 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.689028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.691060 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.691066 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.693281 (+ 2215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.693296 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.694729 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.694735 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.695007 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.695064 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 02:48:56.695209 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 02:48:56.695413 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11305,"mutex_wait_us":1,"spinlock_wait_cycles":9344,"thread_start_us":11163,"threads_started":1}
I20260925 02:48:56.696098 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.675648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.5.56.130:38155)
0925 02:48:56.683869 (+ 8221us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:56.683883 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:56.687179 (+ 3296us) server_negotiation.cc:407] Connection header received
0925 02:48:56.687321 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:56.687334 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:56.687644 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:56.687903 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:56.689242 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.690911 (+ 1669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.693449 (+ 2538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.694517 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.694656 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.694877 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:56.695620 (+ 743us) server_negotiation.cc:299] Negotiation successful
0925 02:48:56.695801 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7971,"thread_start_us":7830,"threads_started":1}
I20260925 02:48:56.696146 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.674516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60206 (local address 127.5.56.131:32777)
0925 02:48:56.687666 (+ 13150us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:56.687677 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:56.687710 (+ 33us) server_negotiation.cc:407] Connection header received
0925 02:48:56.687815 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:56.687822 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:56.688051 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:56.688283 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:56.688975 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.690318 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.693102 (+ 2784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.694552 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.694672 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.694914 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:56.695606 (+ 692us) server_negotiation.cc:299] Negotiation successful
0925 02:48:56.695819 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12958,"thread_start_us":12786,"threads_started":1}
I20260925 02:48:56.696532 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.674952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37590 (local address 127.5.56.129:37753)
0925 02:48:56.688140 (+ 13188us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:56.688149 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:56.688174 (+ 25us) server_negotiation.cc:407] Connection header received
0925 02:48:56.688539 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:56.688557 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:56.688809 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:56.689671 (+ 862us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:56.689888 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.691331 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.693646 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.695052 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.695213 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.695491 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:56.696002 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 02:48:56.696165 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12999,"thread_start_us":12834,"threads_started":1}
I20260925 02:48:56.697041 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.676643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:37753 (local address 127.0.0.1:37590)
0925 02:48:56.686854 (+ 10211us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:56.686874 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:56.687024 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:56.689041 (+ 2017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:56.689048 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:56.689081 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:56.689487 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.689499 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.691479 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.691484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.693483 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.693496 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.695284 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.695291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.695458 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.695493 (+ 35us) client_negotiation.cc:769] Sending connection context
0925 02:48:56.695588 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 02:48:56.695722 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10066,"thread_start_us":7450,"threads_started":1}
I20260925 02:48:56.697243 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.674357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:32777 (local address 127.0.0.1:60206)
0925 02:48:56.684626 (+ 10269us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:56.684699 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:56.684968 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:56.688283 (+ 3315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:56.688288 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:56.688317 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:56.688736 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.688750 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.690499 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.690505 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.692912 (+ 2407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.692927 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.695022 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.695028 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.695244 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.695274 (+ 30us) client_negotiation.cc:769] Sending connection context
0925 02:48:56.695349 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 02:48:56.695456 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9927,"mutex_wait_us":28,"thread_start_us":12170,"threads_started":1}
I20260925 02:48:56.700497 25049 tablet_service.cc:1467] Processing CreateTablet for tablet a58719d40c664ca3904516b8c443b357 (DEFAULT_TABLE table=test-table [id=1a37f461cc2f462a828d58494bcd5655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:56.700551 25184 tablet_service.cc:1467] Processing CreateTablet for tablet a58719d40c664ca3904516b8c443b357 (DEFAULT_TABLE table=test-table [id=1a37f461cc2f462a828d58494bcd5655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:56.700682 25320 tablet_service.cc:1467] Processing CreateTablet for tablet a58719d40c664ca3904516b8c443b357 (DEFAULT_TABLE table=test-table [id=1a37f461cc2f462a828d58494bcd5655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:56.701958 25184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58719d40c664ca3904516b8c443b357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:56.701964 25320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58719d40c664ca3904516b8c443b357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:56.702996 25049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a58719d40c664ca3904516b8c443b357. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:56.714166 25405 tablet_bootstrap.cc:492] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: Bootstrap starting.
I20260925 02:48:56.718623 25405 tablet_bootstrap.cc:654] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:56.718623 25406 tablet_bootstrap.cc:492] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1: Bootstrap starting.
I20260925 02:48:56.720175 25405 log.cc:824] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:56.720498 25407 tablet_bootstrap.cc:492] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: Bootstrap starting.
I20260925 02:48:56.722451 25406 tablet_bootstrap.cc:654] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:56.723790 25406 log.cc:824] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:56.723865 25407 tablet_bootstrap.cc:654] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:56.724360 25405 tablet_bootstrap.cc:492] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: No bootstrap required, opened a new log
I20260925 02:48:56.724740 25405 ts_tablet_manager.cc:1397] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.006s
I20260925 02:48:56.725065 25407 log.cc:824] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:56.728281 25406 tablet_bootstrap.cc:492] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1: No bootstrap required, opened a new log
I20260925 02:48:56.728282 25407 tablet_bootstrap.cc:492] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: No bootstrap required, opened a new log
I20260925 02:48:56.728562 25407 ts_tablet_manager.cc:1397] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260925 02:48:56.728562 25406 ts_tablet_manager.cc:1397] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260925 02:48:56.732728 25405 raft_consensus.cc:348] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7d24fe4aa9455286b162b235126bb1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 32777 } } peers { permanent_uuid: "130a2ae994614e028d717bdb0f53b1d4" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37753 } } peers { permanent_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38155 } }
I20260925 02:48:56.733150 25405 raft_consensus.cc:374] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:56.733291 25405 raft_consensus.cc:729] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 130a2ae994614e028d717bdb0f53b1d4, State: Initialized, Role: FOLLOWER
I20260925 02:48:56.733958 25405 consensus_queue.cc:260] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7d24fe4aa9455286b162b235126bb1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 32777 } } peers { permanent_uuid: "130a2ae994614e028d717bdb0f53b1d4" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37753 } } peers { permanent_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38155 } }
I20260925 02:48:56.736032 25405 ts_tablet_manager.cc:1428] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: Time spent starting tablet: real 0.011s user 0.004s sys 0.007s
I20260925 02:48:56.739567 25406 raft_consensus.cc:348] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7d24fe4aa9455286b162b235126bb1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 32777 } } peers { permanent_uuid: "130a2ae994614e028d717bdb0f53b1d4" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37753 } } peers { permanent_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38155 } }
I20260925 02:48:56.740056 25406 raft_consensus.cc:374] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:56.740208 25406 raft_consensus.cc:729] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d7d24fe4aa9455286b162b235126bb1, State: Initialized, Role: FOLLOWER
I20260925 02:48:56.740051 25407 raft_consensus.cc:348] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7d24fe4aa9455286b162b235126bb1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 32777 } } peers { permanent_uuid: "130a2ae994614e028d717bdb0f53b1d4" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37753 } } peers { permanent_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38155 } }
I20260925 02:48:56.740492 25407 raft_consensus.cc:374] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:56.740644 25407 raft_consensus.cc:729] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da3de2ea6d804451a3f93a0bee35e4d2, State: Initialized, Role: FOLLOWER
W20260925 02:48:56.740890 25115 tablet.cc:2369] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:56.740870 25406 consensus_queue.cc:260] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7d24fe4aa9455286b162b235126bb1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 32777 } } peers { permanent_uuid: "130a2ae994614e028d717bdb0f53b1d4" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37753 } } peers { permanent_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38155 } }
I20260925 02:48:56.741321 25407 consensus_queue.cc:260] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7d24fe4aa9455286b162b235126bb1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 32777 } } peers { permanent_uuid: "130a2ae994614e028d717bdb0f53b1d4" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37753 } } peers { permanent_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38155 } }
I20260925 02:48:56.743243 25385 heartbeater.cc:503] Master 127.5.56.190:43037 was elected leader, sending a full tablet report...
I20260925 02:48:56.743829 25406 ts_tablet_manager.cc:1428] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20260925 02:48:56.744148 25249 heartbeater.cc:503] Master 127.5.56.190:43037 was elected leader, sending a full tablet report...
I20260925 02:48:56.744284 25407 ts_tablet_manager.cc:1428] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: Time spent starting tablet: real 0.016s user 0.008s sys 0.007s
I20260925 02:48:56.755740 25413 raft_consensus.cc:482] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:56.756084 25413 raft_consensus.cc:504] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7d24fe4aa9455286b162b235126bb1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 32777 } } peers { permanent_uuid: "130a2ae994614e028d717bdb0f53b1d4" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37753 } } peers { permanent_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38155 } }
I20260925 02:48:56.757699 25413 leader_election.cc:288] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d7d24fe4aa9455286b162b235126bb1 (127.5.56.131:32777), 130a2ae994614e028d717bdb0f53b1d4 (127.5.56.129:37753)
I20260925 02:48:56.767460 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.758136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:56767 (local address 127.5.56.131:32777)
0925 02:48:56.758355 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:56.758361 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:56.759492 (+ 1131us) server_negotiation.cc:407] Connection header received
0925 02:48:56.759707 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:56.759714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:56.759805 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:56.759916 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:56.760890 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.762033 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.765049 (+ 3016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.766278 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.766498 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.766592 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:56.767055 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 02:48:56.767189 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:48:56.767462 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.758137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:54015 (local address 127.5.56.129:37753)
0925 02:48:56.758345 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:56.758352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:56.759493 (+ 1141us) server_negotiation.cc:407] Connection header received
0925 02:48:56.759731 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:56.759737 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:56.759812 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:56.759916 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:56.760890 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.762033 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.765178 (+ 3145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.766464 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.766613 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.766675 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:56.767054 (+ 379us) server_negotiation.cc:299] Negotiation successful
0925 02:48:56.767190 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:48:56.767679 25415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.758194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:32777 (local address 127.5.56.130:56767)
0925 02:48:56.759359 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:56.759375 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:56.759507 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:56.759945 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:56.759951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:56.760004 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:56.760678 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.760698 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.762179 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.762185 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.764874 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.764892 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.766581 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.766591 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.766847 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.766910 (+ 63us) client_negotiation.cc:769] Sending connection context
0925 02:48:56.767011 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 02:48:56.767182 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1057,"mutex_wait_us":237,"spinlock_wait_cycles":138496,"thread_start_us":554,"threads_started":1}
I20260925 02:48:56.768173 25414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.758188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:37753 (local address 127.5.56.130:54015)
0925 02:48:56.759292 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:56.759313 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:56.759507 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:56.759945 (+ 438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:56.759951 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:56.760004 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:56.760706 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.760716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.762180 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.762185 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.764874 (+ 2689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:56.764892 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.766657 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.766662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:56.766895 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.766919 (+ 24us) client_negotiation.cc:769] Sending connection context
0925 02:48:56.767011 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 02:48:56.767182 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":934,"mutex_wait_us":281,"spinlock_wait_cycles":27520,"thread_start_us":597,"threads_started":1}
I20260925 02:48:56.768858 25340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58719d40c664ca3904516b8c443b357" candidate_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7d24fe4aa9455286b162b235126bb1" is_pre_election: true
I20260925 02:48:56.769137 25069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58719d40c664ca3904516b8c443b357" candidate_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130a2ae994614e028d717bdb0f53b1d4" is_pre_election: true
I20260925 02:48:56.769403 25340 raft_consensus.cc:2454] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3de2ea6d804451a3f93a0bee35e4d2 in term 0.
I20260925 02:48:56.769667 25069 raft_consensus.cc:2454] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate da3de2ea6d804451a3f93a0bee35e4d2 in term 0.
I20260925 02:48:56.770195 25138 leader_election.cc:302] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d7d24fe4aa9455286b162b235126bb1, da3de2ea6d804451a3f93a0bee35e4d2; no voters:
I20260925 02:48:56.770689 25413 raft_consensus.cc:2781] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:56.770864 25413 raft_consensus.cc:482] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:56.770977 25413 raft_consensus.cc:3037] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:56.773211 25413 raft_consensus.cc:504] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7d24fe4aa9455286b162b235126bb1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 32777 } } peers { permanent_uuid: "130a2ae994614e028d717bdb0f53b1d4" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37753 } } peers { permanent_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38155 } }
I20260925 02:48:56.773901 25413 leader_election.cc:288] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [CANDIDATE]: Term 1 election: Requested vote from peers 7d7d24fe4aa9455286b162b235126bb1 (127.5.56.131:32777), 130a2ae994614e028d717bdb0f53b1d4 (127.5.56.129:37753)
I20260925 02:48:56.774417 25340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58719d40c664ca3904516b8c443b357" candidate_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d7d24fe4aa9455286b162b235126bb1"
I20260925 02:48:56.774454 25069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a58719d40c664ca3904516b8c443b357" candidate_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "130a2ae994614e028d717bdb0f53b1d4"
I20260925 02:48:56.774657 25340 raft_consensus.cc:3037] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:56.774667 25069 raft_consensus.cc:3037] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:56.777005 25069 raft_consensus.cc:2454] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3de2ea6d804451a3f93a0bee35e4d2 in term 1.
I20260925 02:48:56.777379 25340 raft_consensus.cc:2454] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate da3de2ea6d804451a3f93a0bee35e4d2 in term 1.
I20260925 02:48:56.777453 25139 leader_election.cc:302] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 130a2ae994614e028d717bdb0f53b1d4, da3de2ea6d804451a3f93a0bee35e4d2; no voters:
I20260925 02:48:56.777832 25413 raft_consensus.cc:2781] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:56.778255 25413 raft_consensus.cc:686] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 1 LEADER]: Becoming Leader. State: Replica: da3de2ea6d804451a3f93a0bee35e4d2, State: Running, Role: LEADER
I20260925 02:48:56.778760 25413 consensus_queue.cc:237] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7d24fe4aa9455286b162b235126bb1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 32777 } } peers { permanent_uuid: "130a2ae994614e028d717bdb0f53b1d4" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37753 } } peers { permanent_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38155 } }
I20260925 02:48:56.783308 24924 catalog_manager.cc:5495] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to da3de2ea6d804451a3f93a0bee35e4d2 (127.5.56.130). New cstate: current_term: 1 leader_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d7d24fe4aa9455286b162b235126bb1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 32777 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "130a2ae994614e028d717bdb0f53b1d4" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37753 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "da3de2ea6d804451a3f93a0bee35e4d2" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 38155 } health_report { overall_health: HEALTHY } } }
W20260925 02:48:56.812747 25250 tablet.cc:2369] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:56.821921 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.816518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47358 (local address 127.5.56.130:38155)
0925 02:48:56.816726 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:56.816732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:56.816753 (+ 21us) server_negotiation.cc:407] Connection header received
0925 02:48:56.816974 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:56.816987 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:56.817075 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:56.817231 (+ 156us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:56.817621 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.818329 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.819727 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.820142 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.820275 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.820781 (+ 506us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:56.821326 (+ 545us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:56.821493 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:56.821555 (+ 62us) server_negotiation.cc:299] Negotiation successful
0925 02:48:56.821640 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:48:56.830297 25340 raft_consensus.cc:1261] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1 [term 1 FOLLOWER]: Refusing update from remote peer da3de2ea6d804451a3f93a0bee35e4d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:56.830292 25069 raft_consensus.cc:1261] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4 [term 1 FOLLOWER]: Refusing update from remote peer da3de2ea6d804451a3f93a0bee35e4d2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:56.831312 25413 consensus_queue.cc:1035] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d7d24fe4aa9455286b162b235126bb1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 32777 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:56.831638 25416 consensus_queue.cc:1035] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "130a2ae994614e028d717bdb0f53b1d4" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 37753 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:56.845218 25419 mvcc.cc:205] Tried to move back new op lower bound from 7333087382847037440 to 7333087382649659392. Current Snapshot: MvccSnapshot[applied={T|T < 7333087382847037440}]
I20260925 02:48:56.848601 25421 mvcc.cc:205] Tried to move back new op lower bound from 7333087382847037440 to 7333087382649659392. Current Snapshot: MvccSnapshot[applied={T|T < 7333087382847037440}]
I20260925 02:48:56.862998 24923 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:56.863359 24923 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48050:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:56.867460 24923 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e092db2a0fe841188d8e9ee0f3228979: Sending DeleteTablet for 3 replicas of tablet a58719d40c664ca3904516b8c443b357
I20260925 02:48:56.868752 25320 tablet_service.cc:1514] Processing DeleteTablet for tablet a58719d40c664ca3904516b8c443b357 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:56 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:60206
I20260925 02:48:56.868857 25184 tablet_service.cc:1514] Processing DeleteTablet for tablet a58719d40c664ca3904516b8c443b357 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:56 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:47342
I20260925 02:48:56.869597 25049 tablet_service.cc:1514] Processing DeleteTablet for tablet a58719d40c664ca3904516b8c443b357 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:56 UTC) from {username='kudu', principal='kudu/127.5.56.190@KRBTEST.COM'} at 127.0.0.1:37590
I20260925 02:48:56.869812 25428 tablet_replica.cc:330] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1: stopping tablet replica
I20260925 02:48:56.870321 25429 tablet_replica.cc:330] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: stopping tablet replica
I20260925 02:48:56.870280 25428 raft_consensus.cc:2229] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:56.870728 25429 raft_consensus.cc:2229] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:56.870847 25428 raft_consensus.cc:2258] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:56.871496 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 24986
I20260925 02:48:56.871500 25429 raft_consensus.cc:2258] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:56.872210 25430 tablet_replica.cc:330] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: stopping tablet replica
I20260925 02:48:56.872723 25430 raft_consensus.cc:2229] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:56.873451 25430 raft_consensus.cc:2258] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:56.874747 25428 ts_tablet_manager.cc:1905] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:56.875752 25430 ts_tablet_manager.cc:1905] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:56.879964 25429 ts_tablet_manager.cc:1905] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:56.883299 25428 ts_tablet_manager.cc:1918] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:56.883530 25428 log.cc:1196] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/a58719d40c664ca3904516b8c443b357
I20260925 02:48:56.884141 25428 ts_tablet_manager.cc:1939] T a58719d40c664ca3904516b8c443b357 P 7d7d24fe4aa9455286b162b235126bb1: Deleting consensus metadata
I20260925 02:48:56.884493 25430 ts_tablet_manager.cc:1918] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:56.884749 25430 log.cc:1196] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-0/wal/wals/a58719d40c664ca3904516b8c443b357
I20260925 02:48:56.885366 25430 ts_tablet_manager.cc:1939] T a58719d40c664ca3904516b8c443b357 P 130a2ae994614e028d717bdb0f53b1d4: Deleting consensus metadata
I20260925 02:48:56.886268 24909 catalog_manager.cc:4841] TS 7d7d24fe4aa9455286b162b235126bb1 (127.5.56.131:32777): tablet a58719d40c664ca3904516b8c443b357 (table test-table [id=1a37f461cc2f462a828d58494bcd5655]) successfully deleted
I20260925 02:48:56.887372 24910 catalog_manager.cc:4841] TS 130a2ae994614e028d717bdb0f53b1d4 (127.5.56.129:37753): tablet a58719d40c664ca3904516b8c443b357 (table test-table [id=1a37f461cc2f462a828d58494bcd5655]) successfully deleted
I20260925 02:48:56.890014 25429 ts_tablet_manager.cc:1918] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:56.890223 25429 log.cc:1196] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/a58719d40c664ca3904516b8c443b357
I20260925 02:48:56.890748 25429 ts_tablet_manager.cc:1939] T a58719d40c664ca3904516b8c443b357 P da3de2ea6d804451a3f93a0bee35e4d2: Deleting consensus metadata
I20260925 02:48:56.892638 24911 catalog_manager.cc:4841] TS da3de2ea6d804451a3f93a0bee35e4d2 (127.5.56.130:38155): tablet a58719d40c664ca3904516b8c443b357 (table test-table [id=1a37f461cc2f462a828d58494bcd5655]) successfully deleted
Sep 25 02:48:56 dist-test-slave-n563 krb5kdc[24879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.129@KRBTEST.COM
I20260925 02:48:56.904702 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:56.873050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37592 (local address 127.5.56.129:37753)
0925 02:48:56.873230 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:56.873240 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:56.887317 (+ 14077us) server_negotiation.cc:407] Connection header received
0925 02:48:56.887598 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:56.887602 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:56.887697 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:56.887789 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:56.889654 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.890837 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.892661 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:56.893362 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:56.893488 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:56.898265 (+ 4777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:56.898281 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:56.898297 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:56.898380 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:56.901459 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:56.902373 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:56.902424 (+ 51us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:56.902430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:56.902572 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:56.903012 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:56.903017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:56.903022 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:56.903472 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:56.903691 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:56.904086 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 02:48:56.904259 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:48:56.905454 25109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:57.027129 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 24986
I20260925 02:48:57.053043 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 25121
Sep 25 02:48:57 dist-test-slave-n563 krb5kdc[24879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.130@KRBTEST.COM
I20260925 02:48:57.072867 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:57.053933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47370 (local address 127.5.56.130:38155)
0925 02:48:57.054179 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:57.054185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:57.054204 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:48:57.054431 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:57.054438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:57.054533 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:57.054643 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:57.056069 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:57.056944 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:57.058364 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:57.059261 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:57.059403 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:57.065104 (+ 5701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:57.065117 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:57.065140 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:57.065210 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:57.069477 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:57.070479 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:57.070489 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:57.070493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:57.070673 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:57.071115 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:57.071120 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:57.071123 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:57.071597 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:57.071947 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:57.072230 (+ 283us) server_negotiation.cc:299] Negotiation successful
0925 02:48:57.072470 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260925 02:48:57.073683 25244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:57.190949 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 25121
I20260925 02:48:57.207743 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 25255
Sep 25 02:48:57 dist-test-slave-n563 krb5kdc[24879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.131@KRBTEST.COM
I20260925 02:48:57.223809 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:57.208584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.5.56.131:32777)
0925 02:48:57.208751 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:57.208755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:57.208832 (+ 77us) server_negotiation.cc:407] Connection header received
0925 02:48:57.209007 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:57.209011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:57.209086 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:57.209172 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:57.210815 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:57.211550 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:57.212936 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:57.213312 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:57.213408 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:57.217972 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:57.217987 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:57.218004 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:48:57.218038 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:57.220874 (+ 2836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:57.221727 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:57.221735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:57.221738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:57.221912 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:48:57.222303 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:48:57.222306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:48:57.222308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:48:57.222794 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:57.223010 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:57.223329 (+ 319us) server_negotiation.cc:299] Negotiation successful
0925 02:48:57.223512 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:48:57.224563 25380 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:57.323428 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 25255
I20260925 02:48:57.341059 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 24892
I20260925 02:48:57.342231 24954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:57.457996 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 24892
2026-09-25T02:48:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4503 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T02:48:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:48:57Z Disabled control of system clock
I20260925 02:48:57.540171 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:34421
--webserver_interface=127.5.56.190
--webserver_port=0
--builtin_ntp_servers=127.5.56.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:34421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:48:57.882687 25439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:57.883006 25439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:57.883088 25439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:57.890352 25439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:48:57.890492 25439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:57.890530 25439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:57.890595 25439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:48:57.890640 25439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:48:57.899981 25439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:34421
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:34421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:57.901296 25439 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:57.903091 25439 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:57.915194 25445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:57.915239 25444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:57.915331 25447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:57.915992 25439 server_base.cc:1034] running on GCE node
I20260925 02:48:57.916837 25439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:57.918359 25439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:57.919663 25439 hybrid_clock.cc:648] HybridClock initialized: now 1790304537919605 us; error 53 us; skew 500 ppm
I20260925 02:48:57.920044 25439 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:57.926438 25439 webserver.cc:460] Webserver started at http://127.5.56.190:38477/ using document root <none> and password file <none>
I20260925 02:48:57.927248 25439 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:57.927372 25439 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:57.927731 25439 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:57.930951 25439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "99be99411a234f578af3b27e036f87ad"
format_stamp: "Formatted at 2026-09-25 02:48:57 on dist-test-slave-n563"
server_key: "3bbcc4d8cabad3503b2b3132a6b912ea"
server_key_iv: "90da9c849d0e60a36048f66636c7224e"
server_key_version: "encryptionkey@0"
I20260925 02:48:57.931808 25439 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "99be99411a234f578af3b27e036f87ad"
format_stamp: "Formatted at 2026-09-25 02:48:57 on dist-test-slave-n563"
server_key: "3bbcc4d8cabad3503b2b3132a6b912ea"
server_key_iv: "90da9c849d0e60a36048f66636c7224e"
server_key_version: "encryptionkey@0"
I20260925 02:48:57.937299 25439 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:48:57.940975 25453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:57.942497 25439 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:48:57.942698 25439 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "99be99411a234f578af3b27e036f87ad"
format_stamp: "Formatted at 2026-09-25 02:48:57 on dist-test-slave-n563"
server_key: "3bbcc4d8cabad3503b2b3132a6b912ea"
server_key_iv: "90da9c849d0e60a36048f66636c7224e"
server_key_version: "encryptionkey@0"
I20260925 02:48:57.942950 25439 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:57.955766 25439 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:57.956677 25439 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:57.956985 25439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:57.976198 25439 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:34421
I20260925 02:48:57.976222 25505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:34421 every 8 connection(s)
I20260925 02:48:57.977626 25439 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:48:57.981724 25506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:57.986519 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 25439
I20260925 02:48:57.986703 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:48:57.987112 5346 external_mini_cluster.cc:1428] Setting key 1196eef2e090f97a11011b188c9338c0
I20260925 02:48:57.994360 25506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad: Bootstrap starting.
I20260925 02:48:57.998204 25506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:57.999605 25506 log.cc:824] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:58.003355 25506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad: No bootstrap required, opened a new log
I20260925 02:48:58.003535 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:57.989166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51622 (local address 127.5.56.190:34421)
0925 02:48:57.990360 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:57.990408 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:57.990472 (+ 64us) server_negotiation.cc:407] Connection header received
0925 02:48:57.992996 (+ 2524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:57.993026 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:57.993636 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:57.994085 (+ 449us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:57.994681 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:57.996738 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:57.998134 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:57.999265 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:57.999478 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:58.000665 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:58.000673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:58.000690 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:58.001417 (+ 727us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:58.001520 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:58.001546 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:58.001671 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:58.001895 (+ 224us) server_negotiation.cc:299] Negotiation successful
0925 02:48:58.002222 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":395,"threads_started":1}
I20260925 02:48:58.011454 25506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99be99411a234f578af3b27e036f87ad" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34421 } }
I20260925 02:48:58.011826 25506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:58.011914 25506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99be99411a234f578af3b27e036f87ad, State: Initialized, Role: FOLLOWER
I20260925 02:48:58.012601 25506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99be99411a234f578af3b27e036f87ad" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34421 } }
I20260925 02:48:58.012881 25506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:48:58.013052 25506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:48:58.013233 25506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:58.015552 25506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99be99411a234f578af3b27e036f87ad" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34421 } }
I20260925 02:48:58.016121 25506 leader_election.cc:302] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99be99411a234f578af3b27e036f87ad; no voters:
I20260925 02:48:58.016721 25506 leader_election.cc:288] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:48:58.016880 25511 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:58.017647 25511 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [term 1 LEADER]: Becoming Leader. State: Replica: 99be99411a234f578af3b27e036f87ad, State: Running, Role: LEADER
I20260925 02:48:58.018141 25511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99be99411a234f578af3b27e036f87ad" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34421 } }
I20260925 02:48:58.018960 25506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:48:58.021100 25512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99be99411a234f578af3b27e036f87ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99be99411a234f578af3b27e036f87ad" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34421 } } }
I20260925 02:48:58.021623 25512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:58.022075 25513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99be99411a234f578af3b27e036f87ad. Latest consensus state: current_term: 1 leader_uuid: "99be99411a234f578af3b27e036f87ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99be99411a234f578af3b27e036f87ad" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34421 } } }
I20260925 02:48:58.022369 25513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad [sys.catalog]: This master's current role is: LEADER
I20260925 02:48:58.023279 25520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:48:58.030052 25520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:48:58.039001 25520 catalog_manager.cc:1347] Generated new cluster ID: 1f18d66d776748d8b246e5db9c4ca5b4
I20260925 02:48:58.039130 25520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:48:58.056128 25520 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:48:58.058109 25520 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:48:58.068173 25520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad: Generated new TSK 0
I20260925 02:48:58.069321 25520 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:48:58.077261 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.129:0
--local_ip_for_outbound_sockets=127.5.56.129
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34421
--builtin_ntp_servers=127.5.56.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 02:48:58.423653 25530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:58.424017 25530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:58.424201 25530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:58.431838 25530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:58.431989 25530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:58.432176 25530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.129
I20260925 02:48:58.443382 25530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.56.129
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.129
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:58.444808 25530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:58.446766 25530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:58.460668 25535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:58.460644 25538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:58.460919 25536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:58.461594 25530 server_base.cc:1034] running on GCE node
I20260925 02:48:58.462265 25530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:58.463330 25530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:58.464576 25530 hybrid_clock.cc:648] HybridClock initialized: now 1790304538464512 us; error 54 us; skew 500 ppm
I20260925 02:48:58.464924 25530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:58.467468 25530 webserver.cc:460] Webserver started at http://127.5.56.129:41119/ using document root <none> and password file <none>
I20260925 02:48:58.468187 25530 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:58.468323 25530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:58.468660 25530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:58.471685 25530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/data/instance:
uuid: "b3fe486f91ff4cd1999d2218b1068e27"
format_stamp: "Formatted at 2026-09-25 02:48:58 on dist-test-slave-n563"
server_key: "7487c79bd993e413680f3de340f3d3e7"
server_key_iv: "0d5e0c000bfe92b24d84f5a794c6cc34"
server_key_version: "encryptionkey@0"
I20260925 02:48:58.472503 25530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance:
uuid: "b3fe486f91ff4cd1999d2218b1068e27"
format_stamp: "Formatted at 2026-09-25 02:48:58 on dist-test-slave-n563"
server_key: "7487c79bd993e413680f3de340f3d3e7"
server_key_iv: "0d5e0c000bfe92b24d84f5a794c6cc34"
server_key_version: "encryptionkey@0"
I20260925 02:48:58.477854 25530 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:48:58.481513 25544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:58.482933 25530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:58.483116 25530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/wal
uuid: "b3fe486f91ff4cd1999d2218b1068e27"
format_stamp: "Formatted at 2026-09-25 02:48:58 on dist-test-slave-n563"
server_key: "7487c79bd993e413680f3de340f3d3e7"
server_key_iv: "0d5e0c000bfe92b24d84f5a794c6cc34"
server_key_version: "encryptionkey@0"
I20260925 02:48:58.483361 25530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:58.499553 25530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:58.500451 25530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:58.500774 25530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:58.501789 25530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:58.503525 25530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:58.503654 25530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:58.503787 25530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:58.503876 25530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:58.538118 25530 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.129:46163
I20260925 02:48:58.538168 25657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.129:46163 every 8 connection(s)
I20260925 02:48:58.539503 25530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/data/info.pb
I20260925 02:48:58.547652 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 25530
I20260925 02:48:58.547833 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-0/wal/instance
I20260925 02:48:58.548373 5346 external_mini_cluster.cc:1428] Setting key 5eadedb1f3b9ce39422517c96ad9f9cd
I20260925 02:48:58.551376 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.130:0
--local_ip_for_outbound_sockets=127.5.56.130
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34421
--builtin_ntp_servers=127.5.56.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:48:58.553313 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:58.542150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.129:42639 (local address 127.5.56.190:34421)
0925 02:48:58.542651 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:58.542655 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:58.543815 (+ 1160us) server_negotiation.cc:407] Connection header received
0925 02:48:58.545625 (+ 1810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:58.545631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:58.545726 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:58.545853 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:58.547103 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:58.548275 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:58.550137 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:58.550700 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:58.550857 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:58.552326 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:58.552336 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:58.552344 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:58.552372 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:58.552430 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:58.552449 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:58.552552 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:58.552767 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 02:48:58.552886 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":287,"threads_started":1}
I20260925 02:48:58.554227 25661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:58.542669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:34421 (local address 127.5.56.129:42639)
0925 02:48:58.543636 (+ 967us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:58.543672 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:58.545347 (+ 1675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:58.546124 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:58.546136 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:58.546247 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:58.546936 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:58.546954 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:58.548459 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:58.548467 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:58.549970 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:58.550009 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:58.550963 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:58.550968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:58.551189 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:58.551995 (+ 806us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:58.551998 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:58.552041 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:58.552128 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:58.552554 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:58.552562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:58.552569 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:48:58.552715 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 02:48:58.553051 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":337,"threads_started":1}
I20260925 02:48:58.555866 25658 heartbeater.cc:347] Connected to a master server at 127.5.56.190:34421
I20260925 02:48:58.556202 25658 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:58.557195 25658 heartbeater.cc:511] Master 127.5.56.190:34421 requested a full tablet report, sending...
I20260925 02:48:58.559499 25471 ts_manager.cc:194] Registered new tserver with Master: b3fe486f91ff4cd1999d2218b1068e27 (127.5.56.129:46163)
I20260925 02:48:58.561096 25471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.129:42639
W20260925 02:48:58.892323 25662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:58.892624 25662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:58.892722 25662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:58.900035 25662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:58.900148 25662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:58.900260 25662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.130
I20260925 02:48:58.909509 25662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.56.130
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.130
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:58.910882 25662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:58.912786 25662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:58.926837 25670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:58.926867 25668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:58.926889 25667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:58.927729 25662 server_base.cc:1034] running on GCE node
I20260925 02:48:58.928457 25662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:58.929526 25662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:58.930771 25662 hybrid_clock.cc:648] HybridClock initialized: now 1790304538930726 us; error 44 us; skew 500 ppm
I20260925 02:48:58.931113 25662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:58.933749 25662 webserver.cc:460] Webserver started at http://127.5.56.130:41611/ using document root <none> and password file <none>
I20260925 02:48:58.934648 25662 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:58.934806 25662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:58.935165 25662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:58.938342 25662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/data/instance:
uuid: "1d0bd35ed5314c15993b272843d82443"
format_stamp: "Formatted at 2026-09-25 02:48:58 on dist-test-slave-n563"
server_key: "958b3db268492434e6062c168bcb41ac"
server_key_iv: "a86c04853131c59fc731689aa949bbf8"
server_key_version: "encryptionkey@0"
I20260925 02:48:58.939325 25662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance:
uuid: "1d0bd35ed5314c15993b272843d82443"
format_stamp: "Formatted at 2026-09-25 02:48:58 on dist-test-slave-n563"
server_key: "958b3db268492434e6062c168bcb41ac"
server_key_iv: "a86c04853131c59fc731689aa949bbf8"
server_key_version: "encryptionkey@0"
I20260925 02:48:58.945056 25662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:48:58.949234 25676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:58.950870 25662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:48:58.951078 25662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/wal
uuid: "1d0bd35ed5314c15993b272843d82443"
format_stamp: "Formatted at 2026-09-25 02:48:58 on dist-test-slave-n563"
server_key: "958b3db268492434e6062c168bcb41ac"
server_key_iv: "a86c04853131c59fc731689aa949bbf8"
server_key_version: "encryptionkey@0"
I20260925 02:48:58.951342 25662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:58.972041 25662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:58.972998 25662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:58.973351 25662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:58.974467 25662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:58.976155 25662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:58.976289 25662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:58.976439 25662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:58.976536 25662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:59.014261 25662 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.130:45395
I20260925 02:48:59.014358 25790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.130:45395 every 8 connection(s)
I20260925 02:48:59.015771 25662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/data/info.pb
I20260925 02:48:59.020784 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 25662
I20260925 02:48:59.020983 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/wal/instance
I20260925 02:48:59.021452 5346 external_mini_cluster.cc:1428] Setting key bfa1179842630e1ecc2c063ca1e16b86
I20260925 02:48:59.024572 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.56.131:0
--local_ip_for_outbound_sockets=127.5.56.131
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34421
--builtin_ntp_servers=127.5.56.148:39397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 02:48:59.030490 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.018692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.130:36475 (local address 127.5.56.190:34421)
0925 02:48:59.018908 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:59.018912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:59.020588 (+ 1676us) server_negotiation.cc:407] Connection header received
0925 02:48:59.022707 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:59.022712 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:59.022785 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:59.022889 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:59.024123 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.025232 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.027075 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.027804 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.027963 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.029474 (+ 1511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:59.029482 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:59.029489 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:59.029518 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:59.029561 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:59.029580 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:59.029677 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:59.029929 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 02:48:59.030044 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:48:59.031473 25793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.019343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:34421 (local address 127.5.56.130:36475)
0925 02:48:59.020418 (+ 1075us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:59.020460 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:59.022228 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:59.023116 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:59.023126 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:59.023259 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:59.023940 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.023960 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.025383 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.025388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.026926 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.026939 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.028030 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.028035 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.028359 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.029128 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:59.029131 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:59.029174 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:59.029269 (+ 95us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:59.029710 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:59.029718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:59.029725 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 02:48:59.029882 (+ 157us) client_negotiation.cc:240] Negotiation successful
0925 02:48:59.030213 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":309,"threads_started":1}
I20260925 02:48:59.033156 25791 heartbeater.cc:347] Connected to a master server at 127.5.56.190:34421
I20260925 02:48:59.033495 25791 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:59.034268 25791 heartbeater.cc:511] Master 127.5.56.190:34421 requested a full tablet report, sending...
I20260925 02:48:59.035887 25471 ts_manager.cc:194] Registered new tserver with Master: 1d0bd35ed5314c15993b272843d82443 (127.5.56.130:45395)
I20260925 02:48:59.037248 25471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.130:36475
W20260925 02:48:59.372067 25794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:48:59.372370 25794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:48:59.372575 25794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:48:59.379904 25794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:48:59.380019 25794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:48:59.380210 25794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.56.131
I20260925 02:48:59.389854 25794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:39397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.56.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.56.131
--webserver_port=0
--tserver_master_addrs=127.5.56.190:34421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.56.131
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:48:59.391255 25794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:48:59.393076 25794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:48:59.406719 25799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:59.406761 25800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:48:59.406957 25802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:48:59.407449 25794 server_base.cc:1034] running on GCE node
I20260925 02:48:59.408061 25794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:48:59.409004 25794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:48:59.410216 25794 hybrid_clock.cc:648] HybridClock initialized: now 1790304539410151 us; error 55 us; skew 500 ppm
I20260925 02:48:59.410614 25794 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260925 02:48:59.412925 25794 webserver.cc:460] Webserver started at http://127.5.56.131:44257/ using document root <none> and password file <none>
I20260925 02:48:59.413647 25794 fs_manager.cc:359] Metadata directory not provided
I20260925 02:48:59.413764 25794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:48:59.414119 25794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:48:59.417157 25794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/data/instance:
uuid: "d441fb7b76e542f58e31542d02c7ecc1"
format_stamp: "Formatted at 2026-09-25 02:48:59 on dist-test-slave-n563"
server_key: "a6fe6cc907d3c1693b7e827ebdd9d55a"
server_key_iv: "9681f8b39ce6433224e1107babd35498"
server_key_version: "encryptionkey@0"
I20260925 02:48:59.417999 25794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance:
uuid: "d441fb7b76e542f58e31542d02c7ecc1"
format_stamp: "Formatted at 2026-09-25 02:48:59 on dist-test-slave-n563"
server_key: "a6fe6cc907d3c1693b7e827ebdd9d55a"
server_key_iv: "9681f8b39ce6433224e1107babd35498"
server_key_version: "encryptionkey@0"
I20260925 02:48:59.423517 25794 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260925 02:48:59.427188 25808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:59.428546 25794 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:48:59.428732 25794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/wal
uuid: "d441fb7b76e542f58e31542d02c7ecc1"
format_stamp: "Formatted at 2026-09-25 02:48:59 on dist-test-slave-n563"
server_key: "a6fe6cc907d3c1693b7e827ebdd9d55a"
server_key_iv: "9681f8b39ce6433224e1107babd35498"
server_key_version: "encryptionkey@0"
I20260925 02:48:59.428973 25794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:48:59.448086 25794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:48:59.449000 25794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:48:59.449321 25794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:48:59.450356 25794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 02:48:59.451906 25794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 02:48:59.452036 25794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:59.452174 25794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 02:48:59.452265 25794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 02:48:59.486320 25794 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.131:40849
I20260925 02:48:59.486356 25921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.131:40849 every 8 connection(s)
I20260925 02:48:59.487808 25794 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/data/info.pb
I20260925 02:48:59.494684 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 25794
I20260925 02:48:59.494863 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/wal/instance
I20260925 02:48:59.495429 5346 external_mini_cluster.cc:1428] Setting key 8cd446e32df9eb431154a85497f3ff70
I20260925 02:48:59.501214 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.490484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:57849 (local address 127.5.56.190:34421)
0925 02:48:59.490712 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:59.490716 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:59.492192 (+ 1476us) server_negotiation.cc:407] Connection header received
0925 02:48:59.493900 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:59.493904 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:59.493976 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:59.494066 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:59.495535 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.496441 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.498663 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.499083 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.499182 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.500539 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:59.500543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:59.500546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:59.500565 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:59.500595 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:59.500606 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:59.500671 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:59.500896 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 02:48:59.500974 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260925 02:48:59.502238 25924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.491133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:34421 (local address 127.5.56.131:57849)
0925 02:48:59.492042 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:59.492083 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:59.493655 (+ 1572us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:59.494323 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:59.494336 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:59.494479 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:48:59.495331 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.495356 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.496629 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.496636 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.498533 (+ 1897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.498546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.499276 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.499281 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.499477 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.500195 (+ 718us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 02:48:59.500198 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:48:59.500240 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 02:48:59.500322 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:48:59.500688 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:48:59.500697 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:48:59.500702 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 02:48:59.500855 (+ 153us) client_negotiation.cc:240] Negotiation successful
0925 02:48:59.501162 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":289,"threads_started":1}
I20260925 02:48:59.503624 25922 heartbeater.cc:347] Connected to a master server at 127.5.56.190:34421
I20260925 02:48:59.503933 25922 heartbeater.cc:464] Registering TS with master...
I20260925 02:48:59.504709 25922 heartbeater.cc:511] Master 127.5.56.190:34421 requested a full tablet report, sending...
I20260925 02:48:59.506135 25471 ts_manager.cc:194] Registered new tserver with Master: d441fb7b76e542f58e31542d02c7ecc1 (127.5.56.131:40849)
I20260925 02:48:59.506956 25471 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.56.131:57849
I20260925 02:48:59.510634 5346 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 02:48:59.521114 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.514953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.5.56.190:34421)
0925 02:48:59.515215 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:59.515222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:59.515576 (+ 354us) server_negotiation.cc:407] Connection header received
0925 02:48:59.515779 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:59.515785 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:59.515871 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:59.515980 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:59.516455 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.517156 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.518668 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.519113 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.519275 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.520381 (+ 1106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:59.520386 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:59.520389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:59.520410 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:59.520444 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:59.520456 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:59.520555 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:59.520703 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 02:48:59.520798 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:48:59.525070 25471 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 02:48:59.532099 25471 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 02:48:59.559706 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.545152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:45395 (local address 127.0.0.1:58790)
0925 02:48:59.546605 (+ 1453us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:59.546646 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:59.546913 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:59.548183 (+ 1270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:59.548194 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:59.548255 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:59.548715 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.548747 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.551397 (+ 2650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.551407 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.554802 (+ 3395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.554827 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.558251 (+ 3424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.558261 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.558643 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.558724 (+ 81us) client_negotiation.cc:769] Sending connection context
0925 02:48:59.558899 (+ 175us) client_negotiation.cc:240] Negotiation successful
0925 02:48:59.559179 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1174,"thread_start_us":994,"threads_started":1}
I20260925 02:48:59.561020 25935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.545374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58790 (local address 127.5.56.130:45395)
0925 02:48:59.545960 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:59.545971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:59.547145 (+ 1174us) server_negotiation.cc:407] Connection header received
0925 02:48:59.547292 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:59.547304 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:59.547622 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:59.547827 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:59.548967 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.551166 (+ 2199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.555186 (+ 4020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.557543 (+ 2357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.557715 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.558069 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:59.560250 (+ 2181us) server_negotiation.cc:299] Negotiation successful
0925 02:48:59.560538 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":297,"threads_started":1}
I20260925 02:48:59.564393 25658 heartbeater.cc:503] Master 127.5.56.190:34421 was elected leader, sending a full tablet report...
I20260925 02:48:59.570160 25724 tablet_service.cc:1467] Processing CreateTablet for tablet e6f91af52c4b45318c7813c896f73081 (DEFAULT_TABLE table=test-table [id=e55ac3c76e9c4804ba0ca6ce50f74696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:59.572145 25724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f91af52c4b45318c7813c896f73081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:59.576819 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.547139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46163 (local address 127.0.0.1:44588)
0925 02:48:59.563447 (+ 16308us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:59.563471 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:59.563627 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:59.565681 (+ 2054us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:59.565689 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:59.565728 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:59.566221 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.566236 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.568854 (+ 2618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.568863 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.573683 (+ 4820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.573700 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.575981 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.575990 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.576180 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.576222 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 02:48:59.576336 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 02:48:59.576479 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16094,"thread_start_us":15913,"threads_started":1}
I20260925 02:48:59.576997 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.547659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44588 (local address 127.5.56.129:46163)
0925 02:48:59.562849 (+ 15190us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:59.562862 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:59.563780 (+ 918us) server_negotiation.cc:407] Connection header received
0925 02:48:59.563923 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:59.563935 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:59.565107 (+ 1172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:59.565390 (+ 283us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:59.566490 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.568614 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.573875 (+ 5261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.575426 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.575572 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.575849 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:59.576412 (+ 563us) server_negotiation.cc:299] Negotiation successful
0925 02:48:59.576607 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14930,"thread_start_us":14771,"threads_started":1}
I20260925 02:48:59.577487 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.545737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.131:40849 (local address 127.0.0.1:55652)
0925 02:48:59.561619 (+ 15882us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:59.561648 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:59.561800 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:59.571542 (+ 9742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:59.571547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:59.571578 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:59.571820 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.571837 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.573787 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.573795 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.575360 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.575373 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.576694 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.576702 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.576934 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.576979 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 02:48:59.577051 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 02:48:59.577214 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15685,"thread_start_us":26145,"threads_started":1}
I20260925 02:48:59.579133 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.546310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.5.56.131:40849)
0925 02:48:59.570944 (+ 24634us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:59.570950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:59.570980 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:48:59.571089 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:59.571100 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:59.571344 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:59.571546 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:59.572061 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.573598 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.575524 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.576509 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.577793 (+ 1284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.578512 (+ 719us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:59.578596 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 02:48:59.578766 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24445,"thread_start_us":24294,"threads_started":1}
I20260925 02:48:59.581009 25592 tablet_service.cc:1467] Processing CreateTablet for tablet e6f91af52c4b45318c7813c896f73081 (DEFAULT_TABLE table=test-table [id=e55ac3c76e9c4804ba0ca6ce50f74696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:59.582391 25592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f91af52c4b45318c7813c896f73081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:59.582803 25856 tablet_service.cc:1467] Processing CreateTablet for tablet e6f91af52c4b45318c7813c896f73081 (DEFAULT_TABLE table=test-table [id=e55ac3c76e9c4804ba0ca6ce50f74696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 02:48:59.584352 25856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6f91af52c4b45318c7813c896f73081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:48:59.588925 25940 tablet_bootstrap.cc:492] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: Bootstrap starting.
I20260925 02:48:59.592793 25940 tablet_bootstrap.cc:654] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:59.594345 25940 log.cc:824] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:59.596412 25942 tablet_bootstrap.cc:492] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27: Bootstrap starting.
I20260925 02:48:59.597731 25940 tablet_bootstrap.cc:492] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: No bootstrap required, opened a new log
I20260925 02:48:59.598073 25940 ts_tablet_manager.cc:1397] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 02:48:59.598220 25943 tablet_bootstrap.cc:492] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: Bootstrap starting.
I20260925 02:48:59.601428 25942 tablet_bootstrap.cc:654] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:59.601832 25943 tablet_bootstrap.cc:654] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: Neither blocks nor log segments found. Creating new log.
I20260925 02:48:59.602880 25942 log.cc:824] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:59.602982 25943 log.cc:824] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:48:59.605823 25942 tablet_bootstrap.cc:492] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27: No bootstrap required, opened a new log
I20260925 02:48:59.606117 25942 ts_tablet_manager.cc:1397] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260925 02:48:59.606248 25943 tablet_bootstrap.cc:492] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: No bootstrap required, opened a new log
I20260925 02:48:59.606544 25943 ts_tablet_manager.cc:1397] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260925 02:48:59.607395 25940 raft_consensus.cc:348] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bd35ed5314c15993b272843d82443" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45395 } } peers { permanent_uuid: "b3fe486f91ff4cd1999d2218b1068e27" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46163 } } peers { permanent_uuid: "d441fb7b76e542f58e31542d02c7ecc1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40849 } }
I20260925 02:48:59.607887 25940 raft_consensus.cc:374] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:59.607999 25940 raft_consensus.cc:729] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d0bd35ed5314c15993b272843d82443, State: Initialized, Role: FOLLOWER
I20260925 02:48:59.608644 25940 consensus_queue.cc:260] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bd35ed5314c15993b272843d82443" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45395 } } peers { permanent_uuid: "b3fe486f91ff4cd1999d2218b1068e27" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46163 } } peers { permanent_uuid: "d441fb7b76e542f58e31542d02c7ecc1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40849 } }
I20260925 02:48:59.611366 25791 heartbeater.cc:503] Master 127.5.56.190:34421 was elected leader, sending a full tablet report...
I20260925 02:48:59.612061 25940 ts_tablet_manager.cc:1428] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: Time spent starting tablet: real 0.014s user 0.005s sys 0.007s
I20260925 02:48:59.616029 25942 raft_consensus.cc:348] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bd35ed5314c15993b272843d82443" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45395 } } peers { permanent_uuid: "b3fe486f91ff4cd1999d2218b1068e27" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46163 } } peers { permanent_uuid: "d441fb7b76e542f58e31542d02c7ecc1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40849 } }
I20260925 02:48:59.616434 25942 raft_consensus.cc:374] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:59.616539 25942 raft_consensus.cc:729] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3fe486f91ff4cd1999d2218b1068e27, State: Initialized, Role: FOLLOWER
I20260925 02:48:59.616300 25943 raft_consensus.cc:348] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bd35ed5314c15993b272843d82443" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45395 } } peers { permanent_uuid: "b3fe486f91ff4cd1999d2218b1068e27" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46163 } } peers { permanent_uuid: "d441fb7b76e542f58e31542d02c7ecc1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40849 } }
I20260925 02:48:59.616760 25943 raft_consensus.cc:374] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:48:59.616966 25943 raft_consensus.cc:729] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d441fb7b76e542f58e31542d02c7ecc1, State: Initialized, Role: FOLLOWER
I20260925 02:48:59.617259 25942 consensus_queue.cc:260] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bd35ed5314c15993b272843d82443" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45395 } } peers { permanent_uuid: "b3fe486f91ff4cd1999d2218b1068e27" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46163 } } peers { permanent_uuid: "d441fb7b76e542f58e31542d02c7ecc1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40849 } }
I20260925 02:48:59.617611 25943 consensus_queue.cc:260] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bd35ed5314c15993b272843d82443" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45395 } } peers { permanent_uuid: "b3fe486f91ff4cd1999d2218b1068e27" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46163 } } peers { permanent_uuid: "d441fb7b76e542f58e31542d02c7ecc1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40849 } }
I20260925 02:48:59.619302 25942 ts_tablet_manager.cc:1428] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260925 02:48:59.620807 25922 heartbeater.cc:503] Master 127.5.56.190:34421 was elected leader, sending a full tablet report...
I20260925 02:48:59.621196 25943 ts_tablet_manager.cc:1428] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
W20260925 02:48:59.739547 25923 tablet.cc:2369] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:59.758365 25948 raft_consensus.cc:482] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:48:59.758780 25948 raft_consensus.cc:504] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bd35ed5314c15993b272843d82443" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45395 } } peers { permanent_uuid: "b3fe486f91ff4cd1999d2218b1068e27" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46163 } } peers { permanent_uuid: "d441fb7b76e542f58e31542d02c7ecc1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40849 } }
I20260925 02:48:59.760543 25948 leader_election.cc:288] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d0bd35ed5314c15993b272843d82443 (127.5.56.130:45395), b3fe486f91ff4cd1999d2218b1068e27 (127.5.56.129:46163)
W20260925 02:48:59.767993 25792 tablet.cc:2369] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:59.769023 25924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.761078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.130:45395 (local address 127.5.56.131:38757)
0925 02:48:59.761315 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:59.761354 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:59.761511 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:59.762179 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:59.762184 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:59.762225 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:59.762487 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.762506 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.763941 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.763946 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.766128 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.766143 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.768061 (+ 1918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.768069 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.768247 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.768293 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 02:48:59.768401 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 02:48:59.768580 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":71}
I20260925 02:48:59.769276 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.761889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:50027 (local address 127.5.56.129:46163)
0925 02:48:59.762113 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:59.762121 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:59.762635 (+ 514us) server_negotiation.cc:407] Connection header received
0925 02:48:59.762809 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:59.762818 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:59.762931 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:59.763059 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:59.763835 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.764871 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.767159 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.767983 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.768148 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.768220 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:59.768896 (+ 676us) server_negotiation.cc:299] Negotiation successful
0925 02:48:59.769047 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:48:59.769946 25935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.761086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.5.56.131:38757 (local address 127.5.56.130:45395)
0925 02:48:59.761296 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:59.761302 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:59.761433 (+ 131us) server_negotiation.cc:407] Connection header received
0925 02:48:59.761716 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:59.761725 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:59.761823 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:59.761915 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 02:48:59.762719 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.763785 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.766324 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.767880 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.768064 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.768150 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:59.769540 (+ 1390us) server_negotiation.cc:299] Negotiation successful
0925 02:48:59.769682 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:48:59.770107 25949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.761561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46163 (local address 127.5.56.131:50027)
0925 02:48:59.762516 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0925 02:48:59.762538 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:48:59.762659 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:48:59.763094 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:48:59.763099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:48:59.763124 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 02:48:59.763542 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.763553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.765058 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.765063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.766967 (+ 1904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:48:59.766978 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.768499 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.768506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:48:59.768696 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.768732 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 02:48:59.769508 (+ 776us) client_negotiation.cc:240] Negotiation successful
0925 02:48:59.769767 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":804,"mutex_wait_us":3,"thread_start_us":609,"threads_started":1}
I20260925 02:48:59.770887 25745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f91af52c4b45318c7813c896f73081" candidate_uuid: "d441fb7b76e542f58e31542d02c7ecc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bd35ed5314c15993b272843d82443" is_pre_election: true
I20260925 02:48:59.771446 25745 raft_consensus.cc:2454] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d441fb7b76e542f58e31542d02c7ecc1 in term 0.
I20260925 02:48:59.771409 25613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f91af52c4b45318c7813c896f73081" candidate_uuid: "d441fb7b76e542f58e31542d02c7ecc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fe486f91ff4cd1999d2218b1068e27" is_pre_election: true
I20260925 02:48:59.772023 25613 raft_consensus.cc:2454] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d441fb7b76e542f58e31542d02c7ecc1 in term 0.
I20260925 02:48:59.772241 25812 leader_election.cc:302] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bd35ed5314c15993b272843d82443, d441fb7b76e542f58e31542d02c7ecc1; no voters:
I20260925 02:48:59.772715 25948 raft_consensus.cc:2781] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 02:48:59.772884 25948 raft_consensus.cc:482] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 02:48:59.773002 25948 raft_consensus.cc:3037] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:59.775429 25948 raft_consensus.cc:504] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bd35ed5314c15993b272843d82443" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45395 } } peers { permanent_uuid: "b3fe486f91ff4cd1999d2218b1068e27" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46163 } } peers { permanent_uuid: "d441fb7b76e542f58e31542d02c7ecc1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40849 } }
I20260925 02:48:59.776106 25948 leader_election.cc:288] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [CANDIDATE]: Term 1 election: Requested vote from peers 1d0bd35ed5314c15993b272843d82443 (127.5.56.130:45395), b3fe486f91ff4cd1999d2218b1068e27 (127.5.56.129:46163)
I20260925 02:48:59.776646 25745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f91af52c4b45318c7813c896f73081" candidate_uuid: "d441fb7b76e542f58e31542d02c7ecc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d0bd35ed5314c15993b272843d82443"
I20260925 02:48:59.776724 25613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6f91af52c4b45318c7813c896f73081" candidate_uuid: "d441fb7b76e542f58e31542d02c7ecc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3fe486f91ff4cd1999d2218b1068e27"
I20260925 02:48:59.776880 25745 raft_consensus.cc:3037] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:59.776983 25613 raft_consensus.cc:3037] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:48:59.779350 25745 raft_consensus.cc:2454] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d441fb7b76e542f58e31542d02c7ecc1 in term 1.
I20260925 02:48:59.779842 25812 leader_election.cc:302] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1d0bd35ed5314c15993b272843d82443, d441fb7b76e542f58e31542d02c7ecc1; no voters:
I20260925 02:48:59.780159 25948 raft_consensus.cc:2781] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:48:59.780763 25948 raft_consensus.cc:686] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 1 LEADER]: Becoming Leader. State: Replica: d441fb7b76e542f58e31542d02c7ecc1, State: Running, Role: LEADER
I20260925 02:48:59.780989 25613 raft_consensus.cc:2454] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d441fb7b76e542f58e31542d02c7ecc1 in term 1.
I20260925 02:48:59.781395 25948 consensus_queue.cc:237] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bd35ed5314c15993b272843d82443" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45395 } } peers { permanent_uuid: "b3fe486f91ff4cd1999d2218b1068e27" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46163 } } peers { permanent_uuid: "d441fb7b76e542f58e31542d02c7ecc1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40849 } }
I20260925 02:48:59.787119 25471 catalog_manager.cc:5495] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d441fb7b76e542f58e31542d02c7ecc1 (127.5.56.131). New cstate: current_term: 1 leader_uuid: "d441fb7b76e542f58e31542d02c7ecc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d0bd35ed5314c15993b272843d82443" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b3fe486f91ff4cd1999d2218b1068e27" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d441fb7b76e542f58e31542d02c7ecc1" member_type: VOTER last_known_addr { host: "127.5.56.131" port: 40849 } health_report { overall_health: HEALTHY } } }
W20260925 02:48:59.791920 25659 tablet.cc:2369] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:48:59.826870 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.819808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55662 (local address 127.5.56.131:40849)
0925 02:48:59.820031 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:59.820040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:59.820177 (+ 137us) server_negotiation.cc:407] Connection header received
0925 02:48:59.820428 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:59.820432 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:59.820514 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:59.820638 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:48:59.821126 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.821867 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.824397 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.824848 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.824943 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.825653 (+ 710us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:48:59.826183 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:48:59.826293 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:59.826499 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 02:48:59.826586 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260925 02:48:59.836391 25745 raft_consensus.cc:1261] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443 [term 1 FOLLOWER]: Refusing update from remote peer d441fb7b76e542f58e31542d02c7ecc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:59.836393 25613 raft_consensus.cc:1261] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27 [term 1 FOLLOWER]: Refusing update from remote peer d441fb7b76e542f58e31542d02c7ecc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:48:59.837477 25948 consensus_queue.cc:1035] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3fe486f91ff4cd1999d2218b1068e27" member_type: VOTER last_known_addr { host: "127.5.56.129" port: 46163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:59.837921 25950 consensus_queue.cc:1035] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d0bd35ed5314c15993b272843d82443" member_type: VOTER last_known_addr { host: "127.5.56.130" port: 45395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:48:59.859163 25953 mvcc.cc:205] Tried to move back new op lower bound from 7333087395157131264 to 7333087394949455872. Current Snapshot: MvccSnapshot[applied={T|T < 7333087395157131264}]
I20260925 02:48:59.878839 25471 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:59.879144 25471 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 02:48:59.883882 25471 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 99be99411a234f578af3b27e036f87ad: Sending DeleteTablet for 3 replicas of tablet e6f91af52c4b45318c7813c896f73081
I20260925 02:48:59.885488 25724 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f91af52c4b45318c7813c896f73081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:59 UTC) from {username='slave'} at 127.0.0.1:58790
I20260925 02:48:59.885756 25592 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f91af52c4b45318c7813c896f73081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:59 UTC) from {username='slave'} at 127.0.0.1:44588
I20260925 02:48:59.886984 25856 tablet_service.cc:1514] Processing DeleteTablet for tablet e6f91af52c4b45318c7813c896f73081 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 02:48:59 UTC) from {username='slave'} at 127.0.0.1:55652
I20260925 02:48:59.890326 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 25530
I20260925 02:48:59.894892 25965 tablet_replica.cc:330] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: stopping tablet replica
I20260925 02:48:59.895406 25965 raft_consensus.cc:2229] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 1 LEADER]: Raft consensus shutting down.
I20260925 02:48:59.896147 25965 raft_consensus.cc:2258] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:59.897576 25964 tablet_replica.cc:330] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27: stopping tablet replica
I20260925 02:48:59.898043 25964 raft_consensus.cc:2229] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:59.899988 25964 raft_consensus.cc:2258] T e6f91af52c4b45318c7813c896f73081 P b3fe486f91ff4cd1999d2218b1068e27 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:59.901310 25963 tablet_replica.cc:330] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: stopping tablet replica
I20260925 02:48:59.901907 25963 raft_consensus.cc:2229] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 02:48:59.902513 25963 raft_consensus.cc:2258] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 02:48:59.904902 25963 ts_tablet_manager.cc:1905] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:59.908445 25965 ts_tablet_manager.cc:1905] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 02:48:59.909057 25939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:48:59.892031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.5.56.129:46163)
0925 02:48:59.892257 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:48:59.892266 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:48:59.900768 (+ 8502us) server_negotiation.cc:407] Connection header received
0925 02:48:59.901022 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:48:59.901029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:48:59.901159 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:48:59.901268 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:48:59.902003 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.903165 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.905018 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:48:59.905616 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:48:59.905718 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:48:59.907285 (+ 1567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:48:59.907297 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:48:59.907314 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:48:59.908093 (+ 779us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:48:59.908151 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:48:59.908176 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:48:59.908276 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:48:59.908522 (+ 246us) server_negotiation.cc:299] Negotiation successful
0925 02:48:59.908651 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:48:59.910617 25653 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:48:59.916958 25965 ts_tablet_manager.cc:1918] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:59.917217 25965 log.cc:1196] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-2/wal/wals/e6f91af52c4b45318c7813c896f73081
I20260925 02:48:59.917862 25965 ts_tablet_manager.cc:1939] T e6f91af52c4b45318c7813c896f73081 P d441fb7b76e542f58e31542d02c7ecc1: Deleting consensus metadata
I20260925 02:48:59.920177 25963 ts_tablet_manager.cc:1918] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 02:48:59.920719 25456 catalog_manager.cc:4841] TS d441fb7b76e542f58e31542d02c7ecc1 (127.5.56.131:40849): tablet e6f91af52c4b45318c7813c896f73081 (table test-table [id=e55ac3c76e9c4804ba0ca6ce50f74696]) successfully deleted
I20260925 02:48:59.922570 25963 log.cc:1196] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: Deleting WAL directory at /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790304212743187-5346-0/minicluster-data/ts-1/wal/wals/e6f91af52c4b45318c7813c896f73081
I20260925 02:48:59.923272 25963 ts_tablet_manager.cc:1939] T e6f91af52c4b45318c7813c896f73081 P 1d0bd35ed5314c15993b272843d82443: Deleting consensus metadata
I20260925 02:48:59.925359 25457 catalog_manager.cc:4841] TS 1d0bd35ed5314c15993b272843d82443 (127.5.56.130:45395): tablet e6f91af52c4b45318c7813c896f73081 (table test-table [id=e55ac3c76e9c4804ba0ca6ce50f74696]) successfully deleted
I20260925 02:49:00.037588 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 25530
W20260925 02:49:00.064323 25457 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.5.56.129:46163 (error 108)
I20260925 02:49:00.064492 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 25662
I20260925 02:49:00.066299 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:00.065799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46163 (local address 127.0.0.1:44616)
0925 02:49:00.066036 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:00.066139 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 02:49:00.066963 25457 catalog_manager.cc:4584] TS b3fe486f91ff4cd1999d2218b1068e27 (127.5.56.129:46163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e6f91af52c4b45318c7813c896f73081: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46163: connect: Connection refused (error 111)
I20260925 02:49:00.072337 25935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:00.065871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58806 (local address 127.5.56.130:45395)
0925 02:49:00.066059 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:00.066064 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:00.066081 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:49:00.066137 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:00.066142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:00.066243 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:00.066367 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:00.066829 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:00.067735 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:00.069059 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:00.069491 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:00.069636 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:00.070882 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:00.070891 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:00.070905 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:00.071504 (+ 599us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:00.071553 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:00.071572 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:00.071720 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:00.071853 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 02:49:00.071963 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 02:49:00.072995 25786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:00.120615 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:00.120119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46163 (local address 127.0.0.1:44626)
0925 02:49:00.120367 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:00.120466 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260925 02:49:00.121790 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:00.121387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46163 (local address 127.0.0.1:44640)
0925 02:49:00.121534 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:00.121621 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260925 02:49:00.182705 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:00.182104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46163 (local address 127.0.0.1:44656)
0925 02:49:00.182374 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:00.182556 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260925 02:49:00.183835 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:00.183453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46163 (local address 127.0.0.1:44662)
0925 02:49:00.183611 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:00.183692 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 02:49:00.214711 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 25662
I20260925 02:49:00.233897 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 25794
I20260925 02:49:00.242185 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:00.234766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.5.56.131:40849)
0925 02:49:00.234975 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:00.234981 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:00.235007 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:49:00.235204 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:00.235212 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:00.235355 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:00.235493 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:00.235843 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:00.237100 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:00.238486 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:00.239171 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:00.239327 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:00.240383 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:00.240394 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:00.240414 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:00.241324 (+ 910us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:00.241391 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:00.241417 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:00.241529 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:00.241664 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 02:49:00.241779 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260925 02:49:00.243058 25917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:00.264144 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:00.263652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46163 (local address 127.0.0.1:44664)
0925 02:49:00.263888 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:00.263987 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260925 02:49:00.265343 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:00.264887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46163 (local address 127.0.0.1:44666)
0925 02:49:00.265111 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:00.265181 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260925 02:49:00.369125 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 25794
I20260925 02:49:00.390360 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 25439
I20260925 02:49:00.391528 25501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:00.503592 25934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:00.503034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.129:46163 (local address 127.0.0.1:44672)
0925 02:49:00.503227 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:00.503400 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.129:46163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260925 02:49:00.503620 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 25439
2026-09-25T02:49:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3049 ms)
[----------] 6 tests from EncryptionPolicyTest (22644 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:00 dist-test-slave-n563 krb5kdc[25974](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:00 dist-test-slave-n563 krb5kdc[25974](info): set up 2 sockets
Sep 25 02:49:00 dist-test-slave-n563 krb5kdc[25974](info): commencing operation
krb5kdc: starting...
W20260925 02:49:01.640003 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:01 dist-test-slave-n563 krb5kdc[25974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:01.918051 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42531
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42531
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:02.262195 25987 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:02.262601 25987 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:02.262777 25987 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:02.270152 25987 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:02.270324 25987 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:02.270452 25987 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:02.270567 25987 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:02.270664 25987 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:02.281008 25987 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42531
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:02.282469 25987 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:02.283812 25987 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:02.295588 25992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:02.295543 25993 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:02.295599 25995 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:02.296496 25987 server_base.cc:1034] running on GCE node
I20260925 02:49:02.297340 25987 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:02.298863 25987 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:02.300091 25987 hybrid_clock.cc:648] HybridClock initialized: now 1790304542300037 us; error 49 us; skew 500 ppm
I20260925 02:49:02.300467 25987 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:02 dist-test-slave-n563 krb5kdc[25974](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:02.303728 25987 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:02.305337 25987 webserver.cc:460] Webserver started at http://127.0.0.1:44765/ using document root <none> and password file <none>
I20260925 02:49:02.306243 25987 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:02.306457 25987 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:02.306815 25987 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:02.309998 25987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "ffeb07c72618456c93327f3c8eec64d3"
format_stamp: "Formatted at 2026-09-25 02:49:02 on dist-test-slave-n563"
server_key: "a2be44046b5ed1ff568bb7926a307626"
server_key_iv: "e329705ab3f1ef6acbda306aa499060e"
server_key_version: "encryptionkey@0"
I20260925 02:49:02.310824 25987 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "ffeb07c72618456c93327f3c8eec64d3"
format_stamp: "Formatted at 2026-09-25 02:49:02 on dist-test-slave-n563"
server_key: "a2be44046b5ed1ff568bb7926a307626"
server_key_iv: "e329705ab3f1ef6acbda306aa499060e"
server_key_version: "encryptionkey@0"
I20260925 02:49:02.316218 25987 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:49:02.319726 26002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:02.321043 25987 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:49:02.321249 25987 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "ffeb07c72618456c93327f3c8eec64d3"
format_stamp: "Formatted at 2026-09-25 02:49:02 on dist-test-slave-n563"
server_key: "a2be44046b5ed1ff568bb7926a307626"
server_key_iv: "e329705ab3f1ef6acbda306aa499060e"
server_key_version: "encryptionkey@0"
I20260925 02:49:02.321463 25987 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:02.345700 25987 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:02.349862 25987 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:02.350180 25987 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:02.369874 25987 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42531
I20260925 02:49:02.369920 26053 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42531 every 8 connection(s)
I20260925 02:49:02.371400 25987 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:02.375296 26054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:02.375886 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 25987
I20260925 02:49:02.376104 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:02.376760 5346 external_mini_cluster.cc:1428] Setting key 88946e2e4174fbd57ca19db8401a5c0c
I20260925 02:49:02.388159 26054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3: Bootstrap starting.
Sep 25 02:49:02 dist-test-slave-n563 krb5kdc[25974](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:49:02.391773 26054 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:02.393137 26054 log.cc:824] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:02.396943 26054 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3: No bootstrap required, opened a new log
I20260925 02:49:02.401757 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:02.378807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57061 (local address 127.0.0.1:42531)
0925 02:49:02.379626 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:02.379639 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:02.379680 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:49:02.381726 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:02.381752 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:02.382174 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:02.382647 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:02.384180 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:02.385518 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:02.386908 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:02.387585 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:02.387818 (+ 233us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:02.392471 (+ 4653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:02.392482 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:02.392503 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:02.392553 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:02.396603 (+ 4050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:02.397540 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:02.397550 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:02.397564 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:02.397809 (+ 245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:02.398303 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:02.398308 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:02.398311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:02.399072 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:02.399435 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:02.399914 (+ 479us) server_negotiation.cc:299] Negotiation successful
0925 02:49:02.400302 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":365,"threads_started":1}
I20260925 02:49:02.406219 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:02.407276 26054 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffeb07c72618456c93327f3c8eec64d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42531 } }
I20260925 02:49:02.407657 26054 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:02.407775 26054 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffeb07c72618456c93327f3c8eec64d3, State: Initialized, Role: FOLLOWER
I20260925 02:49:02.408406 26054 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffeb07c72618456c93327f3c8eec64d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42531 } }
I20260925 02:49:02.408640 26054 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:02.408767 26054 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:02.408937 26054 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:02.411368 26054 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffeb07c72618456c93327f3c8eec64d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42531 } }
I20260925 02:49:02.411963 26054 leader_election.cc:302] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffeb07c72618456c93327f3c8eec64d3; no voters:
I20260925 02:49:02.412673 26054 leader_election.cc:288] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:02.412799 26064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:02.414675 26064 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [term 1 LEADER]: Becoming Leader. State: Replica: ffeb07c72618456c93327f3c8eec64d3, State: Running, Role: LEADER
I20260925 02:49:02.415076 26054 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:02.415412 26064 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffeb07c72618456c93327f3c8eec64d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42531 } }
I20260925 02:49:02.419747 26066 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffeb07c72618456c93327f3c8eec64d3. Latest consensus state: current_term: 1 leader_uuid: "ffeb07c72618456c93327f3c8eec64d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffeb07c72618456c93327f3c8eec64d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42531 } } }
I20260925 02:49:02.420177 26066 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:02.420855 26065 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffeb07c72618456c93327f3c8eec64d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffeb07c72618456c93327f3c8eec64d3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42531 } } }
I20260925 02:49:02.421131 26065 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:02.423492 26074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:02.430203 26074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:02.432648 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:02.411523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52645 (local address 127.0.0.1:42531)
0925 02:49:02.411703 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:02.411709 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:02.412204 (+ 495us) server_negotiation.cc:407] Connection header received
0925 02:49:02.412385 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:02.412393 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:02.412506 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:02.412639 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:02.414856 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:02.416089 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:02.418335 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:02.419301 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:02.419479 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:02.423765 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:02.423772 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:02.423777 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:02.423815 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:02.428030 (+ 4215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:02.429186 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:02.429192 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:02.429195 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:02.429375 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:02.430031 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:02.430038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:02.430042 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:02.430515 (+ 473us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:02.430749 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:02.432099 (+ 1350us) server_negotiation.cc:299] Negotiation successful
0925 02:49:02.432284 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:49:02.443655 26074 catalog_manager.cc:1347] Generated new cluster ID: f229f0ac23b543e787982127996d6605
I20260925 02:49:02.443785 26074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:02.462278 26074 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:02.464216 26074 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:02.473596 26074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffeb07c72618456c93327f3c8eec64d3: Generated new TSK 0
I20260925 02:49:02.474601 26074 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:02.487308 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 25987
I20260925 02:49:02.488620 26049 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:02.582314 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 25987
W20260925 02:49:02.600739 25972 connection.cc:535] client connection to 127.0.0.1:42531 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42531): Connection reset by peer (error 104)
2026-09-25T02:49:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2068 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:02 dist-test-slave-n563 krb5kdc[26087](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:02 dist-test-slave-n563 krb5kdc[26087](info): set up 2 sockets
Sep 25 02:49:02 dist-test-slave-n563 krb5kdc[26087](info): commencing operation
krb5kdc: starting...
W20260925 02:49:03.707998 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:03 dist-test-slave-n563 krb5kdc[26087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:03.985598 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:04.328460 26100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:04.328837 26100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:04.329001 26100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:04.336436 26100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:04.336599 26100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:04.336706 26100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:04.336828 26100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:04.336907 26100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:04.347023 26100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:04.348418 26100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:04.349789 26100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:04.361856 26105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:04.361901 26106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:04.361918 26108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:04.363121 26100 server_base.cc:1034] running on GCE node
I20260925 02:49:04.363867 26100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:04.365309 26100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:04.366614 26100 hybrid_clock.cc:648] HybridClock initialized: now 1790304544366572 us; error 55 us; skew 500 ppm
I20260925 02:49:04.366940 26100 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:04 dist-test-slave-n563 krb5kdc[26087](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304544, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:04.375990 26100 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:04.377701 26100 webserver.cc:460] Webserver started at http://127.0.0.1:42033/ using document root <none> and password file <none>
I20260925 02:49:04.378541 26100 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:04.378695 26100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:04.379030 26100 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:04.382236 26100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "3b4d0f1e2f6349739347c94d388ecdd1"
format_stamp: "Formatted at 2026-09-25 02:49:04 on dist-test-slave-n563"
server_key: "f8684744d3b142ebf100d917dee19fe7"
server_key_iv: "d513732962b0bf719218660d213084f6"
server_key_version: "encryptionkey@0"
I20260925 02:49:04.383107 26100 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "3b4d0f1e2f6349739347c94d388ecdd1"
format_stamp: "Formatted at 2026-09-25 02:49:04 on dist-test-slave-n563"
server_key: "f8684744d3b142ebf100d917dee19fe7"
server_key_iv: "d513732962b0bf719218660d213084f6"
server_key_version: "encryptionkey@0"
I20260925 02:49:04.388409 26100 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:49:04.391981 26115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:04.393286 26100 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260925 02:49:04.393476 26100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "3b4d0f1e2f6349739347c94d388ecdd1"
format_stamp: "Formatted at 2026-09-25 02:49:04 on dist-test-slave-n563"
server_key: "f8684744d3b142ebf100d917dee19fe7"
server_key_iv: "d513732962b0bf719218660d213084f6"
server_key_version: "encryptionkey@0"
I20260925 02:49:04.393676 26100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:04.422057 26100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:04.426700 26100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:04.427018 26100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:04.446352 26100 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40627
I20260925 02:49:04.446375 26166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40627 every 8 connection(s)
I20260925 02:49:04.447813 26100 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:04.451850 26167 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:04.453336 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 26100
I20260925 02:49:04.453528 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:04.454095 5346 external_mini_cluster.cc:1428] Setting key d2426d6ef99b68c1db2af33df4cbb5cd
I20260925 02:49:04.466207 26167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1: Bootstrap starting.
I20260925 02:49:04.468355 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:04.456007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34019 (local address 127.0.0.1:40627)
0925 02:49:04.456914 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:04.456930 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:04.456981 (+ 51us) server_negotiation.cc:407] Connection header received
0925 02:49:04.459326 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:04.459357 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:04.459919 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:04.460327 (+ 408us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:04.460879 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:04.462130 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:04.463657 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:04.464393 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:04.464604 (+ 211us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:04.465941 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:04.465949 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:04.465964 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:04.466622 (+ 658us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:04.466701 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:04.466730 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:04.466904 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:04.467184 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 02:49:04.467361 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":408,"threads_started":1}
I20260925 02:49:04.469663 26167 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:04.470963 26167 log.cc:824] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:04.472507 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:04.473928 26167 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1: No bootstrap required, opened a new log
I20260925 02:49:04.483619 26167 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4d0f1e2f6349739347c94d388ecdd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40627 } }
I20260925 02:49:04.484203 26167 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:04.484344 26167 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4d0f1e2f6349739347c94d388ecdd1, State: Initialized, Role: FOLLOWER
I20260925 02:49:04.484477 26170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:04.477279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36991 (local address 127.0.0.1:40627)
0925 02:49:04.477492 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:04.477498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:04.477775 (+ 277us) server_negotiation.cc:407] Connection header received
0925 02:49:04.477952 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:04.477956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:04.478015 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:04.478102 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:04.478772 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:04.479566 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:04.481585 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:04.482056 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:04.482156 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:04.483453 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:04.483458 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:04.483461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:04.483480 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:04.483566 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:04.483584 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:04.483731 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:04.483964 (+ 233us) server_negotiation.cc:299] Negotiation successful
0925 02:49:04.484045 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260925 02:49:04.485190 26167 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4d0f1e2f6349739347c94d388ecdd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40627 } }
I20260925 02:49:04.485546 26167 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:04.485703 26167 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:04.485931 26167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:04.488309 26167 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4d0f1e2f6349739347c94d388ecdd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40627 } }
I20260925 02:49:04.488843 26167 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4d0f1e2f6349739347c94d388ecdd1; no voters:
I20260925 02:49:04.489455 26167 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:04.489575 26177 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:04.490360 26177 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b4d0f1e2f6349739347c94d388ecdd1, State: Running, Role: LEADER
I20260925 02:49:04.490880 26177 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4d0f1e2f6349739347c94d388ecdd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40627 } }
I20260925 02:49:04.491743 26167 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:04.494282 26178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b4d0f1e2f6349739347c94d388ecdd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4d0f1e2f6349739347c94d388ecdd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40627 } } }
I20260925 02:49:04.494180 26179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4d0f1e2f6349739347c94d388ecdd1. Latest consensus state: current_term: 1 leader_uuid: "3b4d0f1e2f6349739347c94d388ecdd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4d0f1e2f6349739347c94d388ecdd1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40627 } } }
I20260925 02:49:04.494688 26178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:04.494688 26179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:04.504201 26185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:04.509483 26185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:04.519414 26185 catalog_manager.cc:1347] Generated new cluster ID: d3d2bedc31ce4e908568b9da618c2453
I20260925 02:49:04.519547 26185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:04.541729 26185 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:04.543195 26185 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:04.551164 26185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b4d0f1e2f6349739347c94d388ecdd1: Generated new TSK 0
I20260925 02:49:04.552206 26185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:04.569413 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26100
W20260925 02:49:04.570775 26162 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:34019
W20260925 02:49:04.571556 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26100: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:04.571748 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 26100
W20260925 02:49:04.589952 26085 connection.cc:535] client connection to 127.0.0.1:40627 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40627): Connection reset by peer (error 104)
2026-09-25T02:49:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1988 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:04 dist-test-slave-n563 krb5kdc[26198](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:04 dist-test-slave-n563 krb5kdc[26198](info): set up 2 sockets
Sep 25 02:49:04 dist-test-slave-n563 krb5kdc[26198](info): commencing operation
krb5kdc: starting...
W20260925 02:49:05.704878 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:05 dist-test-slave-n563 krb5kdc[26198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:05.978273 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46371
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:06.327786 26211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:06.328186 26211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:06.328358 26211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:06.336027 26211 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:06.336210 26211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:06.336334 26211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:06.336449 26211 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:06.336534 26211 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:06.347150 26211 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46371
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:06.348598 26211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:06.349982 26211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:06.362437 26216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:06.362465 26217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:06.363021 26219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:06.363490 26211 server_base.cc:1034] running on GCE node
I20260925 02:49:06.364215 26211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:06.365983 26211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:06.367206 26211 hybrid_clock.cc:648] HybridClock initialized: now 1790304546367158 us; error 43 us; skew 500 ppm
I20260925 02:49:06.367569 26211 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:06 dist-test-slave-n563 krb5kdc[26198](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:06.370790 26211 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:06.372371 26211 webserver.cc:460] Webserver started at http://127.0.0.1:40295/ using document root <none> and password file <none>
I20260925 02:49:06.373085 26211 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:06.373219 26211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:06.373551 26211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:06.376649 26211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "9c6835347d9a47c5919a7f877bea8151"
format_stamp: "Formatted at 2026-09-25 02:49:06 on dist-test-slave-n563"
server_key: "e59e5b9b4631e64ff1fb91ecee64edae"
server_key_iv: "9fd244077e8a3dfec6aa9f31ab4e1f30"
server_key_version: "encryptionkey@0"
I20260925 02:49:06.377418 26211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "9c6835347d9a47c5919a7f877bea8151"
format_stamp: "Formatted at 2026-09-25 02:49:06 on dist-test-slave-n563"
server_key: "e59e5b9b4631e64ff1fb91ecee64edae"
server_key_iv: "9fd244077e8a3dfec6aa9f31ab4e1f30"
server_key_version: "encryptionkey@0"
I20260925 02:49:06.382915 26211 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:49:06.386961 26226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:06.388418 26211 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:49:06.388623 26211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "9c6835347d9a47c5919a7f877bea8151"
format_stamp: "Formatted at 2026-09-25 02:49:06 on dist-test-slave-n563"
server_key: "e59e5b9b4631e64ff1fb91ecee64edae"
server_key_iv: "9fd244077e8a3dfec6aa9f31ab4e1f30"
server_key_version: "encryptionkey@0"
I20260925 02:49:06.388859 26211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:06.394882 26211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:06.399459 26211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:06.399803 26211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:06.419795 26211 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46371
I20260925 02:49:06.419833 26279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46371 every 8 connection(s)
I20260925 02:49:06.421350 26211 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:06.425158 26280 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:06.428606 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 26211
I20260925 02:49:06.428812 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:06.429407 5346 external_mini_cluster.cc:1428] Setting key cfb471b16c1bcc65dbd1bbc6c44ec784
I20260925 02:49:06.437314 26283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:06.431287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40587 (local address 127.0.0.1:46371)
0925 02:49:06.432098 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:06.432110 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:06.432153 (+ 43us) server_negotiation.cc:407] Connection header received
0925 02:49:06.433845 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:06.433869 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:06.434230 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:06.434633 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:06.435728 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:06.435734 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:06.435760 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:06.435777 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:06.435840 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:06.435859 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:06.436031 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:06.436171 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 02:49:06.436372 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":389,"threads_started":1}
I20260925 02:49:06.438015 26280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151: Bootstrap starting.
I20260925 02:49:06.441341 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:06.441896 26280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:06.443177 26280 log.cc:824] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:06.446689 26280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151: No bootstrap required, opened a new log
I20260925 02:49:06.449992 26283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:06.446704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34965 (local address 127.0.0.1:46371)
0925 02:49:06.446904 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:06.446909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:06.448090 (+ 1181us) server_negotiation.cc:407] Connection header received
0925 02:49:06.448207 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:06.448212 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:06.448276 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:06.448400 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:06.449270 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:06.449281 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:06.449286 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:06.449309 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:06.449377 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:06.449398 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:06.449521 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:06.449594 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 02:49:06.449655 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:49:06.455691 26280 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6835347d9a47c5919a7f877bea8151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260925 02:49:06.456076 26280 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:06.456152 26280 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c6835347d9a47c5919a7f877bea8151, State: Initialized, Role: FOLLOWER
I20260925 02:49:06.456739 26280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6835347d9a47c5919a7f877bea8151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260925 02:49:06.456957 26280 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:06.457059 26280 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:06.457182 26280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:06.459643 26280 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6835347d9a47c5919a7f877bea8151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260925 02:49:06.460109 26280 leader_election.cc:302] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c6835347d9a47c5919a7f877bea8151; no voters:
I20260925 02:49:06.460613 26280 leader_election.cc:288] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:06.460768 26290 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:06.461448 26290 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [term 1 LEADER]: Becoming Leader. State: Replica: 9c6835347d9a47c5919a7f877bea8151, State: Running, Role: LEADER
I20260925 02:49:06.461946 26290 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6835347d9a47c5919a7f877bea8151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } }
I20260925 02:49:06.462805 26280 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:06.464974 26291 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c6835347d9a47c5919a7f877bea8151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6835347d9a47c5919a7f877bea8151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } } }
I20260925 02:49:06.465353 26291 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:06.466171 26292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c6835347d9a47c5919a7f877bea8151. Latest consensus state: current_term: 1 leader_uuid: "9c6835347d9a47c5919a7f877bea8151" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c6835347d9a47c5919a7f877bea8151" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46371 } } }
I20260925 02:49:06.466507 26292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:06.468824 26295 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:06.474946 26295 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:06.484383 26295 catalog_manager.cc:1347] Generated new cluster ID: 1e343435feb04313a104ebbdc47d5360
I20260925 02:49:06.484516 26295 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:06.500191 26295 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:06.500627 26295 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:06.509790 26295 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c6835347d9a47c5919a7f877bea8151: Generated new TSK 0
I20260925 02:49:06.510715 26295 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:06.522285 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26211
W20260925 02:49:06.523448 26274 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:40587
W20260925 02:49:06.524313 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26211: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:06.524482 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 26211
2026-09-25T02:49:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1952 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:06 dist-test-slave-n563 krb5kdc[26311](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:06 dist-test-slave-n563 krb5kdc[26311](info): set up 2 sockets
Sep 25 02:49:06 dist-test-slave-n563 krb5kdc[26311](info): commencing operation
krb5kdc: starting...
W20260925 02:49:07.648934 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:07 dist-test-slave-n563 krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:07.921432 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:08.263373 26324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:08.263710 26324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:08.263885 26324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:08.271369 26324 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:08.271494 26324 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:08.271607 26324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:08.271719 26324 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:08.271801 26324 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:08.281896 26324 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41965
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:08.283336 26324 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:08.284680 26324 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:08.296866 26332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:08.297016 26329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:08.296954 26330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:08.297839 26324 server_base.cc:1034] running on GCE node
I20260925 02:49:08.298655 26324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:08.300076 26324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:08.301275 26324 hybrid_clock.cc:648] HybridClock initialized: now 1790304548301235 us; error 27 us; skew 500 ppm
I20260925 02:49:08.301636 26324 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:08 dist-test-slave-n563 krb5kdc[26311](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304548, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:08.304821 26324 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:08.306496 26324 webserver.cc:460] Webserver started at http://127.0.0.1:33157/ using document root <none> and password file <none>
I20260925 02:49:08.307235 26324 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:08.307369 26324 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:08.307693 26324 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:08.310868 26324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "4ea573e99763481f872aa98363a84fe0"
format_stamp: "Formatted at 2026-09-25 02:49:08 on dist-test-slave-n563"
server_key: "80fb51e85c1f481092326bd173efcda7"
server_key_iv: "d7b0f30041acdd3423c0044bb7f3670c"
server_key_version: "encryptionkey@0"
I20260925 02:49:08.311648 26324 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "4ea573e99763481f872aa98363a84fe0"
format_stamp: "Formatted at 2026-09-25 02:49:08 on dist-test-slave-n563"
server_key: "80fb51e85c1f481092326bd173efcda7"
server_key_iv: "d7b0f30041acdd3423c0044bb7f3670c"
server_key_version: "encryptionkey@0"
I20260925 02:49:08.317020 26324 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260925 02:49:08.320605 26339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:08.321988 26324 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:49:08.322183 26324 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "4ea573e99763481f872aa98363a84fe0"
format_stamp: "Formatted at 2026-09-25 02:49:08 on dist-test-slave-n563"
server_key: "80fb51e85c1f481092326bd173efcda7"
server_key_iv: "d7b0f30041acdd3423c0044bb7f3670c"
server_key_version: "encryptionkey@0"
I20260925 02:49:08.322391 26324 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:08.342705 26324 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:08.346994 26324 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:08.347301 26324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:08.366231 26324 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41965
I20260925 02:49:08.366267 26391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41965 every 8 connection(s)
I20260925 02:49:08.367748 26324 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:08.371577 26392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:08.378191 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 26324
I20260925 02:49:08.378477 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:08.378911 5346 external_mini_cluster.cc:1428] Setting key aad17bc27635623ab81841fb59c5e78d
I20260925 02:49:08.382525 26392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0: Bootstrap starting.
I20260925 02:49:08.385676 26392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:08.386974 26392 log.cc:824] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:08.390090 26392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0: No bootstrap required, opened a new log
Sep 25 02:49:08 dist-test-slave-n563 krb5kdc[26311](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:49:08.399793 26392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea573e99763481f872aa98363a84fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260925 02:49:08.400295 26392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:08.400435 26392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea573e99763481f872aa98363a84fe0, State: Initialized, Role: FOLLOWER
I20260925 02:49:08.401163 26392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea573e99763481f872aa98363a84fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260925 02:49:08.401496 26392 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:08.401644 26392 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:08.401880 26392 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:08.402987 26395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:08.380763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36673 (local address 127.0.0.1:41965)
0925 02:49:08.381528 (+ 765us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:08.381541 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:08.381582 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:49:08.383452 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:08.383477 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:08.383888 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:08.384289 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:08.385813 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:08.387340 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:08.389079 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:08.389753 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:08.389894 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:08.394701 (+ 4807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:08.394710 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:08.394725 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:08.394775 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:08.397976 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:08.398986 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:08.398998 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:08.399011 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:08.399285 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:08.399753 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:08.399757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:08.399759 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:08.400476 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:08.400781 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:08.401267 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 02:49:08.401747 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":342,"threads_started":1}
I20260925 02:49:08.404636 26392 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea573e99763481f872aa98363a84fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260925 02:49:08.405220 26392 leader_election.cc:302] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ea573e99763481f872aa98363a84fe0; no voters:
I20260925 02:49:08.405952 26392 leader_election.cc:288] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:08.406080 26397 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:08.407155 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:08.407272 26397 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea573e99763481f872aa98363a84fe0, State: Running, Role: LEADER
I20260925 02:49:08.407896 26397 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea573e99763481f872aa98363a84fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } }
I20260925 02:49:08.408389 26392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:08.410938 26398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ea573e99763481f872aa98363a84fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea573e99763481f872aa98363a84fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } } }
I20260925 02:49:08.411595 26398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:08.411168 26399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ea573e99763481f872aa98363a84fe0. Latest consensus state: current_term: 1 leader_uuid: "4ea573e99763481f872aa98363a84fe0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ea573e99763481f872aa98363a84fe0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41965 } } }
I20260925 02:49:08.415572 26399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:08.423254 26403 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:08.430168 26403 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:08.446650 26403 catalog_manager.cc:1347] Generated new cluster ID: c68c365bea014d678991dae86d8c9874
I20260925 02:49:08.446790 26403 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:08.458810 26395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:08.424511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53955 (local address 127.0.0.1:41965)
0925 02:49:08.424713 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:08.424722 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:08.436054 (+ 11332us) server_negotiation.cc:407] Connection header received
0925 02:49:08.436426 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:08.436436 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:08.436553 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:08.436647 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:08.439279 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:08.440694 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:08.443096 (+ 2402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:08.443567 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:08.443667 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:08.449294 (+ 5627us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:08.449305 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:08.449313 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:08.449385 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:08.454287 (+ 4902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:08.455539 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:08.455549 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:08.455555 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:08.455774 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:08.456369 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:08.456377 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:08.456383 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:08.456902 (+ 519us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:08.457206 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:08.457964 (+ 758us) server_negotiation.cc:299] Negotiation successful
0925 02:49:08.458298 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:49:08.506839 26403 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:08.508252 26403 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:08.515417 26403 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ea573e99763481f872aa98363a84fe0: Generated new TSK 0
I20260925 02:49:08.516475 26403 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:08.531656 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26324
I20260925 02:49:08.532953 26387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:08.630030 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 26324
2026-09-25T02:49:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2106 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:08 dist-test-slave-n563 krb5kdc[26425](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:08 dist-test-slave-n563 krb5kdc[26425](info): set up 2 sockets
Sep 25 02:49:08 dist-test-slave-n563 krb5kdc[26425](info): commencing operation
krb5kdc: starting...
W20260925 02:49:09.743813 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:09 dist-test-slave-n563 krb5kdc[26425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:10.017736 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41819
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46315
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:10.354643 26438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:10.355020 26438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:10.355196 26438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:10.362493 26438 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:10.362668 26438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:10.362783 26438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:10.362888 26438 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:10.362974 26438 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:10.373147 26438 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46315
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41819
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:10.374595 26438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:10.375945 26438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:10.388254 26443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:10.388146 26444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:10.388186 26446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:10.388886 26438 server_base.cc:1034] running on GCE node
I20260925 02:49:10.389796 26438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:10.391350 26438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:10.392588 26438 hybrid_clock.cc:648] HybridClock initialized: now 1790304550392529 us; error 52 us; skew 500 ppm
I20260925 02:49:10.393057 26438 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:10 dist-test-slave-n563 krb5kdc[26425](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:10.401587 26438 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:10.403210 26438 webserver.cc:460] Webserver started at http://127.0.0.1:34653/ using document root <none> and password file <none>
I20260925 02:49:10.403949 26438 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:10.404103 26438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:10.404433 26438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:10.407600 26438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "7afdde357dc246d5b0af86d621078795"
format_stamp: "Formatted at 2026-09-25 02:49:10 on dist-test-slave-n563"
server_key: "bb2e12e2139b3db4cc901a39e52bdd4b"
server_key_iv: "ec35132322eadd21208dc2b05d72b48b"
server_key_version: "encryptionkey@0"
I20260925 02:49:10.408382 26438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "7afdde357dc246d5b0af86d621078795"
format_stamp: "Formatted at 2026-09-25 02:49:10 on dist-test-slave-n563"
server_key: "bb2e12e2139b3db4cc901a39e52bdd4b"
server_key_iv: "ec35132322eadd21208dc2b05d72b48b"
server_key_version: "encryptionkey@0"
I20260925 02:49:10.413813 26438 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:49:10.417372 26453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:10.418761 26438 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260925 02:49:10.418959 26438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "7afdde357dc246d5b0af86d621078795"
format_stamp: "Formatted at 2026-09-25 02:49:10 on dist-test-slave-n563"
server_key: "bb2e12e2139b3db4cc901a39e52bdd4b"
server_key_iv: "ec35132322eadd21208dc2b05d72b48b"
server_key_version: "encryptionkey@0"
I20260925 02:49:10.419173 26438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:10.437050 26438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:10.441128 26438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:10.441442 26438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:10.460373 26438 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41819
I20260925 02:49:10.460397 26505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41819 every 8 connection(s)
I20260925 02:49:10.461845 26438 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:10.465081 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 26438
I20260925 02:49:10.465274 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:10.465864 5346 external_mini_cluster.cc:1428] Setting key 910438c839b1179ee6ba3013cf01f761
I20260925 02:49:10.466159 26506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:10.480207 26509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:10.467695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44057 (local address 127.0.0.1:41819)
0925 02:49:10.468491 (+ 796us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:10.468502 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:10.468543 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:49:10.470951 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:10.470972 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:10.471384 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:10.471717 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:10.472337 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:10.473663 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:10.475593 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:10.476304 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:10.476451 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:10.477768 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:10.477779 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:10.477795 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:10.478445 (+ 650us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:10.478540 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:10.478565 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:10.478664 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:10.478898 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 02:49:10.479181 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":350,"threads_started":1}
I20260925 02:49:10.480783 26506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795: Bootstrap starting.
I20260925 02:49:10.484995 26506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:10.484997 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:10.486069 26506 log.cc:824] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:10.489456 26506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795: No bootstrap required, opened a new log
I20260925 02:49:10.496553 26509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:10.489707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60105 (local address 127.0.0.1:41819)
0925 02:49:10.489886 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:10.489890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:10.490874 (+ 984us) server_negotiation.cc:407] Connection header received
0925 02:49:10.491098 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:10.491103 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:10.491175 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:10.491282 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:10.491834 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:10.492597 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:10.493932 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:10.494489 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:10.494646 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:10.495777 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:10.495783 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:10.495785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:10.495809 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:10.495846 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:10.495864 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:10.495968 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:10.496185 (+ 217us) server_negotiation.cc:299] Negotiation successful
0925 02:49:10.496300 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:49:10.498102 26506 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afdde357dc246d5b0af86d621078795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260925 02:49:10.498677 26506 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:10.498780 26506 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7afdde357dc246d5b0af86d621078795, State: Initialized, Role: FOLLOWER
I20260925 02:49:10.499307 26506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afdde357dc246d5b0af86d621078795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260925 02:49:10.499514 26506 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:10.499612 26506 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:10.499734 26506 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:10.501916 26506 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afdde357dc246d5b0af86d621078795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260925 02:49:10.502454 26506 leader_election.cc:302] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7afdde357dc246d5b0af86d621078795; no voters:
I20260925 02:49:10.503060 26506 leader_election.cc:288] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:10.503201 26516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:10.503912 26516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [term 1 LEADER]: Becoming Leader. State: Replica: 7afdde357dc246d5b0af86d621078795, State: Running, Role: LEADER
I20260925 02:49:10.504528 26516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afdde357dc246d5b0af86d621078795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } }
I20260925 02:49:10.505435 26506 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:10.508074 26518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7afdde357dc246d5b0af86d621078795. Latest consensus state: current_term: 1 leader_uuid: "7afdde357dc246d5b0af86d621078795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afdde357dc246d5b0af86d621078795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } } }
I20260925 02:49:10.508540 26518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:10.509059 26517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7afdde357dc246d5b0af86d621078795" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7afdde357dc246d5b0af86d621078795" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41819 } } }
I20260925 02:49:10.509348 26517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:10.512369 26525 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:10.517915 26525 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:10.527819 26525 catalog_manager.cc:1347] Generated new cluster ID: acdfdc8bc3f7442495a0088d9871f6bd
I20260925 02:49:10.527966 26525 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:10.542330 26525 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:10.543797 26525 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:10.552913 26525 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7afdde357dc246d5b0af86d621078795: Generated new TSK 0
I20260925 02:49:10.554131 26525 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:10.569718 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26438
W20260925 02:49:10.570959 26501 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44057
W20260925 02:49:10.571745 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26438: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:10.571907 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 26438
2026-09-25T02:49:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1941 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:10 dist-test-slave-n563 krb5kdc[26537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:10 dist-test-slave-n563 krb5kdc[26537](info): set up 2 sockets
Sep 25 02:49:10 dist-test-slave-n563 krb5kdc[26537](info): commencing operation
krb5kdc: starting...
W20260925 02:49:11.687865 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:11 dist-test-slave-n563 krb5kdc[26537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:11.961995 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35353
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35353
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:12.298981 26551 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:12.299273 26551 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:12.299363 26551 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:12.306870 26551 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:12.306991 26551 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:12.307037 26551 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:12.307099 26551 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:12.307143 26551 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:12.317022 26551 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35353
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35353
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:12.318331 26551 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:12.319835 26551 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:12.332161 26557 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:12.332244 26556 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:12.332334 26559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:12.333353 26551 server_base.cc:1034] running on GCE node
I20260925 02:49:12.334066 26551 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:12.335531 26551 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:12.336738 26551 hybrid_clock.cc:648] HybridClock initialized: now 1790304552336700 us; error 27 us; skew 500 ppm
I20260925 02:49:12.337138 26551 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:12 dist-test-slave-n563 krb5kdc[26537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304552, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:12.340265 26551 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:12.341873 26551 webserver.cc:460] Webserver started at http://127.0.0.1:38717/ using document root <none> and password file <none>
I20260925 02:49:12.342654 26551 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:12.342805 26551 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:12.343124 26551 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:12.346153 26551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "5cdb9d645fa4463688ea6c0c1859d437"
format_stamp: "Formatted at 2026-09-25 02:49:12 on dist-test-slave-n563"
server_key: "87871134bef8d6445dd04b01f0c9d8b8"
server_key_iv: "8781e1b61ea68f61bbe2b1116daaf494"
server_key_version: "encryptionkey@0"
I20260925 02:49:12.346971 26551 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "5cdb9d645fa4463688ea6c0c1859d437"
format_stamp: "Formatted at 2026-09-25 02:49:12 on dist-test-slave-n563"
server_key: "87871134bef8d6445dd04b01f0c9d8b8"
server_key_iv: "8781e1b61ea68f61bbe2b1116daaf494"
server_key_version: "encryptionkey@0"
I20260925 02:49:12.352097 26551 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260925 02:49:12.355651 26566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:12.356937 26551 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:49:12.357118 26551 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "5cdb9d645fa4463688ea6c0c1859d437"
format_stamp: "Formatted at 2026-09-25 02:49:12 on dist-test-slave-n563"
server_key: "87871134bef8d6445dd04b01f0c9d8b8"
server_key_iv: "8781e1b61ea68f61bbe2b1116daaf494"
server_key_version: "encryptionkey@0"
I20260925 02:49:12.357316 26551 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:12.362939 26551 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:12.367026 26551 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:12.367331 26551 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:12.385916 26551 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35353
I20260925 02:49:12.385962 26617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35353 every 8 connection(s)
I20260925 02:49:12.387439 26551 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:12.388128 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 26551
I20260925 02:49:12.388329 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:12.388917 5346 external_mini_cluster.cc:1428] Setting key adad3b1e94d2fc6e77fa612bdae3f292
I20260925 02:49:12.391956 26618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:12.396737 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:12.390825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47603 (local address 127.0.0.1:35353)
0925 02:49:12.391613 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:12.391625 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:12.391671 (+ 46us) server_negotiation.cc:407] Connection header received
0925 02:49:12.393307 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:12.393329 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:12.393650 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:12.394019 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:12.395140 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:12.395146 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:12.395159 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:12.395172 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:12.395236 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:12.395261 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:12.395508 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:12.395655 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 02:49:12.395836 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":342,"threads_started":1}
I20260925 02:49:12.400545 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:12.403916 26618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437: Bootstrap starting.
I20260925 02:49:12.407229 26618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:12.407598 26621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:12.405111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41817 (local address 127.0.0.1:35353)
0925 02:49:12.405317 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:12.405323 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:12.405555 (+ 232us) server_negotiation.cc:407] Connection header received
0925 02:49:12.405755 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:12.405765 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:12.405826 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:12.405926 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:12.406884 (+ 958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:12.406890 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:12.406894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:12.406911 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:12.406959 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:12.406974 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:12.407090 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:12.407214 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 02:49:12.407287 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:49:12.408504 26618 log.cc:824] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:12.411360 26618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437: No bootstrap required, opened a new log
I20260925 02:49:12.418889 26618 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb9d645fa4463688ea6c0c1859d437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } }
I20260925 02:49:12.419313 26618 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:12.419423 26618 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cdb9d645fa4463688ea6c0c1859d437, State: Initialized, Role: FOLLOWER
I20260925 02:49:12.420022 26618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb9d645fa4463688ea6c0c1859d437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } }
I20260925 02:49:12.420269 26618 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:12.420421 26618 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:12.420594 26618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:12.423110 26618 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb9d645fa4463688ea6c0c1859d437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } }
I20260925 02:49:12.423617 26618 leader_election.cc:302] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cdb9d645fa4463688ea6c0c1859d437; no voters:
I20260925 02:49:12.424219 26618 leader_election.cc:288] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:12.424371 26628 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:12.425156 26628 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [term 1 LEADER]: Becoming Leader. State: Replica: 5cdb9d645fa4463688ea6c0c1859d437, State: Running, Role: LEADER
I20260925 02:49:12.425746 26628 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb9d645fa4463688ea6c0c1859d437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } }
I20260925 02:49:12.426482 26618 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:12.429015 26630 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cdb9d645fa4463688ea6c0c1859d437. Latest consensus state: current_term: 1 leader_uuid: "5cdb9d645fa4463688ea6c0c1859d437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb9d645fa4463688ea6c0c1859d437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } } }
I20260925 02:49:12.429065 26629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cdb9d645fa4463688ea6c0c1859d437" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cdb9d645fa4463688ea6c0c1859d437" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35353 } } }
I20260925 02:49:12.429426 26630 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:12.429483 26629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:12.432159 26636 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:12.437558 26636 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:12.446528 26636 catalog_manager.cc:1347] Generated new cluster ID: 408ec170ec044ec1ac314a4acc73f6c3
I20260925 02:49:12.446676 26636 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:12.460624 26636 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:12.461163 26636 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:12.476895 26636 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cdb9d645fa4463688ea6c0c1859d437: Generated new TSK 0
I20260925 02:49:12.478094 26636 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:12.483835 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26551
W20260925 02:49:12.484947 26613 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47603
W20260925 02:49:12.485769 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26551: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:12.485941 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 26551
2026-09-25T02:49:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1914 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:12 dist-test-slave-n563 krb5kdc[26649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:12 dist-test-slave-n563 krb5kdc[26649](info): set up 2 sockets
Sep 25 02:49:12 dist-test-slave-n563 krb5kdc[26649](info): commencing operation
krb5kdc: starting...
W20260925 02:49:13.624505 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:13 dist-test-slave-n563 krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:13.898265 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39555
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39555
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:14.239796 26662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:14.240152 26662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:14.240331 26662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:14.247812 26662 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:14.247968 26662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:14.248075 26662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:14.248180 26662 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:14.248266 26662 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:14.258361 26662 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39555
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39555
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:14.259824 26662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:14.261161 26662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:14.273196 26668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:14.273355 26667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:14.273272 26670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:14.274358 26662 server_base.cc:1034] running on GCE node
I20260925 02:49:14.275254 26662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:14.276742 26662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:14.278085 26662 hybrid_clock.cc:648] HybridClock initialized: now 1790304554278034 us; error 39 us; skew 500 ppm
I20260925 02:49:14.278561 26662 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:14 dist-test-slave-n563 krb5kdc[26649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304554, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:14.282194 26662 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:14.284783 26662 webserver.cc:460] Webserver started at http://127.0.0.1:38027/ using document root <none> and password file <none>
I20260925 02:49:14.285512 26662 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:14.285652 26662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:14.285975 26662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:14.289036 26662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "9b19202a2b924e69a2d8232066bf25ac"
format_stamp: "Formatted at 2026-09-25 02:49:14 on dist-test-slave-n563"
server_key: "32be16d6c7082141499110dedec57106"
server_key_iv: "069a8cdc73b932fdd1cd3aa7b195713c"
server_key_version: "encryptionkey@0"
I20260925 02:49:14.289934 26662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "9b19202a2b924e69a2d8232066bf25ac"
format_stamp: "Formatted at 2026-09-25 02:49:14 on dist-test-slave-n563"
server_key: "32be16d6c7082141499110dedec57106"
server_key_iv: "069a8cdc73b932fdd1cd3aa7b195713c"
server_key_version: "encryptionkey@0"
I20260925 02:49:14.295383 26662 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:49:14.298987 26677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:14.300319 26662 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:49:14.300508 26662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "9b19202a2b924e69a2d8232066bf25ac"
format_stamp: "Formatted at 2026-09-25 02:49:14 on dist-test-slave-n563"
server_key: "32be16d6c7082141499110dedec57106"
server_key_iv: "069a8cdc73b932fdd1cd3aa7b195713c"
server_key_version: "encryptionkey@0"
I20260925 02:49:14.300721 26662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:14.316573 26662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:14.321022 26662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:14.321341 26662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:14.339998 26662 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39555
I20260925 02:49:14.340030 26729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39555 every 8 connection(s)
I20260925 02:49:14.341439 26662 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:14.345408 26730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:14.345613 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 26662
I20260925 02:49:14.345803 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:14.346371 5346 external_mini_cluster.cc:1428] Setting key 18943cfced220b6b63bb3af4f4ef5b2c
I20260925 02:49:14.359587 26730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac: Bootstrap starting.
Sep 25 02:49:14 dist-test-slave-n563 krb5kdc[26649](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:49:14.363364 26730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:14.364667 26730 log.cc:824] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:14.367871 26730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac: No bootstrap required, opened a new log
I20260925 02:49:14.370908 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:14.348261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:42565 (local address 127.0.0.1:39555)
0925 02:49:14.349143 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:14.349156 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:14.349197 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:49:14.351635 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:14.351657 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:14.352077 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:14.352404 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:14.354166 (+ 1762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:14.355472 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:14.357337 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:14.357986 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:14.358108 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:14.362926 (+ 4818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:14.362934 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:14.362950 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:14.363000 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:14.366623 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:14.367412 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:14.367422 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:14.367434 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:14.367590 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:14.368065 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:14.368071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:14.368074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:14.368627 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:14.368963 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:14.369493 (+ 530us) server_negotiation.cc:299] Negotiation successful
0925 02:49:14.369806 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":432,"threads_started":1}
I20260925 02:49:14.374907 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:14.376791 26730 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b19202a2b924e69a2d8232066bf25ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } }
I20260925 02:49:14.377308 26730 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:14.377421 26730 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b19202a2b924e69a2d8232066bf25ac, State: Initialized, Role: FOLLOWER
I20260925 02:49:14.378126 26730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b19202a2b924e69a2d8232066bf25ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } }
I20260925 02:49:14.378499 26730 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:14.378674 26730 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:14.378901 26730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:14.382275 26730 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b19202a2b924e69a2d8232066bf25ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } }
I20260925 02:49:14.382988 26730 leader_election.cc:302] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b19202a2b924e69a2d8232066bf25ac; no voters:
I20260925 02:49:14.383756 26730 leader_election.cc:288] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:14.383908 26740 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:14.386124 26730 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:14.386718 26740 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [term 1 LEADER]: Becoming Leader. State: Replica: 9b19202a2b924e69a2d8232066bf25ac, State: Running, Role: LEADER
I20260925 02:49:14.387320 26740 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b19202a2b924e69a2d8232066bf25ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } }
I20260925 02:49:14.390638 26741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b19202a2b924e69a2d8232066bf25ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b19202a2b924e69a2d8232066bf25ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } } }
I20260925 02:49:14.391037 26741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:14.391436 26742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b19202a2b924e69a2d8232066bf25ac. Latest consensus state: current_term: 1 leader_uuid: "9b19202a2b924e69a2d8232066bf25ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b19202a2b924e69a2d8232066bf25ac" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39555 } } }
I20260925 02:49:14.391772 26742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:14.394428 26746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:14.399902 26746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:14.401021 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:14.380541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:54481 (local address 127.0.0.1:39555)
0925 02:49:14.380821 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:14.380827 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:14.381141 (+ 314us) server_negotiation.cc:407] Connection header received
0925 02:49:14.381348 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:14.381352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:14.381416 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:14.381489 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:14.383508 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:14.385103 (+ 1595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:14.387024 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:14.387867 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:14.388034 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:14.391719 (+ 3685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:14.391728 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:14.391732 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:14.391789 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:14.395537 (+ 3748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:14.396686 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:14.396695 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:14.396701 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:14.396901 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:14.397671 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:14.397679 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:14.397684 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:14.398244 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:14.398641 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:14.400198 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0925 02:49:14.400466 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:49:14.411505 26746 catalog_manager.cc:1347] Generated new cluster ID: d092d79d706149f3b15d1e152e4f102c
I20260925 02:49:14.411636 26746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:14.428581 26746 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:14.429941 26746 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:14.441370 26746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b19202a2b924e69a2d8232066bf25ac: Generated new TSK 0
I20260925 02:49:14.442261 26746 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:14.451164 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26662
I20260925 02:49:14.452425 26725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:14.546501 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 26662
W20260925 02:49:14.564464 26647 connection.cc:535] client connection to 127.0.0.1:39555 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39555): Connection reset by peer (error 104)
2026-09-25T02:49:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2059 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:14 dist-test-slave-n563 krb5kdc[26763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:14 dist-test-slave-n563 krb5kdc[26763](info): set up 2 sockets
Sep 25 02:49:14 dist-test-slave-n563 krb5kdc[26763](info): commencing operation
krb5kdc: starting...
W20260925 02:49:15.664003 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:15 dist-test-slave-n563 krb5kdc[26763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:15.957940 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:16.298873 26776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:16.299168 26776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:16.299364 26776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:16.306705 26776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:16.306854 26776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:16.306962 26776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:16.307070 26776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:16.307149 26776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:16.317387 26776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:16.318825 26776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:16.320178 26776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:16.332785 26781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:16.332762 26782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:16.332706 26784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:16.333621 26776 server_base.cc:1034] running on GCE node
I20260925 02:49:16.334556 26776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:16.336001 26776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:16.337245 26776 hybrid_clock.cc:648] HybridClock initialized: now 1790304556337203 us; error 37 us; skew 500 ppm
I20260925 02:49:16.337635 26776 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:16 dist-test-slave-n563 krb5kdc[26763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304556, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:16.340893 26776 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:16.342422 26776 webserver.cc:460] Webserver started at http://127.0.0.1:37967/ using document root <none> and password file <none>
I20260925 02:49:16.343137 26776 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:16.343278 26776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:16.343597 26776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:16.347225 26776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "97c9ae2fd6584bbc96932d333b1081e7"
format_stamp: "Formatted at 2026-09-25 02:49:16 on dist-test-slave-n563"
server_key: "f8e1874a423eaee4b185a2ea97fa6843"
server_key_iv: "df4d511d4d0ac1569d5f917a0cb15d58"
server_key_version: "encryptionkey@0"
I20260925 02:49:16.348008 26776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "97c9ae2fd6584bbc96932d333b1081e7"
format_stamp: "Formatted at 2026-09-25 02:49:16 on dist-test-slave-n563"
server_key: "f8e1874a423eaee4b185a2ea97fa6843"
server_key_iv: "df4d511d4d0ac1569d5f917a0cb15d58"
server_key_version: "encryptionkey@0"
I20260925 02:49:16.353314 26776 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:49:16.356966 26791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:16.358345 26776 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:49:16.358628 26776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "97c9ae2fd6584bbc96932d333b1081e7"
format_stamp: "Formatted at 2026-09-25 02:49:16 on dist-test-slave-n563"
server_key: "f8e1874a423eaee4b185a2ea97fa6843"
server_key_iv: "df4d511d4d0ac1569d5f917a0cb15d58"
server_key_version: "encryptionkey@0"
I20260925 02:49:16.358834 26776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:16.404114 26776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:16.408347 26776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:16.408654 26776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:16.427654 26776 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36077
I20260925 02:49:16.427686 26842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36077 every 8 connection(s)
I20260925 02:49:16.429024 26776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:16.432955 26843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:16.435638 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 26776
I20260925 02:49:16.435845 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:16.436300 5346 external_mini_cluster.cc:1428] Setting key d2cbad60681484ce9baf88c0bdd04269
I20260925 02:49:16.445003 26843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7: Bootstrap starting.
I20260925 02:49:16.448774 26843 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:16.449958 26843 log.cc:824] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:16.450178 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:16.438163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:46465 (local address 127.0.0.1:36077)
0925 02:49:16.439014 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:16.439026 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:16.439071 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:49:16.440759 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:16.440783 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:16.441208 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:16.441571 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:16.442219 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:16.443632 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:16.445468 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:16.446127 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:16.446251 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:16.447699 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:16.447707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:16.447721 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:16.448255 (+ 534us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:16.448346 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:16.448371 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:16.448484 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:16.448811 (+ 327us) server_negotiation.cc:299] Negotiation successful
0925 02:49:16.449016 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":417,"threads_started":1}
I20260925 02:49:16.452914 26843 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7: No bootstrap required, opened a new log
I20260925 02:49:16.454015 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:16.463708 26843 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9ae2fd6584bbc96932d333b1081e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } }
I20260925 02:49:16.464157 26843 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:16.464291 26843 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97c9ae2fd6584bbc96932d333b1081e7, State: Initialized, Role: FOLLOWER
I20260925 02:49:16.465009 26843 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9ae2fd6584bbc96932d333b1081e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } }
I20260925 02:49:16.465312 26843 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:16.465494 26843 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:16.465696 26843 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:16.466616 26846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:16.459358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:34775 (local address 127.0.0.1:36077)
0925 02:49:16.459575 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:16.459586 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:16.459902 (+ 316us) server_negotiation.cc:407] Connection header received
0925 02:49:16.460103 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:16.460107 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:16.460178 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:16.460256 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:16.460871 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:16.461662 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:16.463550 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:16.464041 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:16.464177 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:16.465479 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:16.465487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:16.465490 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:16.465510 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:16.465563 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:16.465598 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:16.465713 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:16.466182 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 02:49:16.466238 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260925 02:49:16.468335 26843 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9ae2fd6584bbc96932d333b1081e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } }
I20260925 02:49:16.468789 26843 leader_election.cc:302] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97c9ae2fd6584bbc96932d333b1081e7; no voters:
I20260925 02:49:16.469288 26843 leader_election.cc:288] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:16.469478 26853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:16.470160 26853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [term 1 LEADER]: Becoming Leader. State: Replica: 97c9ae2fd6584bbc96932d333b1081e7, State: Running, Role: LEADER
I20260925 02:49:16.470708 26853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9ae2fd6584bbc96932d333b1081e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } }
I20260925 02:49:16.471473 26843 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:16.473796 26855 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97c9ae2fd6584bbc96932d333b1081e7. Latest consensus state: current_term: 1 leader_uuid: "97c9ae2fd6584bbc96932d333b1081e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9ae2fd6584bbc96932d333b1081e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } } }
I20260925 02:49:16.473766 26854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97c9ae2fd6584bbc96932d333b1081e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97c9ae2fd6584bbc96932d333b1081e7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36077 } } }
I20260925 02:49:16.474293 26855 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:16.474318 26854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:16.479300 26861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:16.484848 26861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:16.494107 26861 catalog_manager.cc:1347] Generated new cluster ID: 802b94275b1c425faa905650060cfa32
I20260925 02:49:16.494239 26861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:16.516868 26861 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:16.518850 26861 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:16.529806 26861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97c9ae2fd6584bbc96932d333b1081e7: Generated new TSK 0
I20260925 02:49:16.530980 26861 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:16.539597 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26776
W20260925 02:49:16.540915 26838 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.130:46465
W20260925 02:49:16.541688 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26776: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:16.541855 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 26776
W20260925 02:49:16.557153 26761 connection.cc:535] client connection to 127.0.0.1:36077 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36077): Connection reset by peer (error 104)
2026-09-25T02:49:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1992 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:16 dist-test-slave-n563 krb5kdc[26875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:16 dist-test-slave-n563 krb5kdc[26875](info): set up 2 sockets
Sep 25 02:49:16 dist-test-slave-n563 krb5kdc[26875](info): commencing operation
krb5kdc: starting...
W20260925 02:49:17.664100 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.088s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:17 dist-test-slave-n563 krb5kdc[26875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:17.945402 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41003
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41003
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:18.283501 26888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:18.283842 26888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:18.284000 26888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:18.291404 26888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:18.291582 26888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:18.291679 26888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:18.291800 26888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:18.291883 26888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:18.302227 26888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:18.303635 26888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:18.304986 26888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:18.317131 26893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:18.317174 26894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:18.317091 26896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:18.318018 26888 server_base.cc:1034] running on GCE node
I20260925 02:49:18.318936 26888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:18.320416 26888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:18.321651 26888 hybrid_clock.cc:648] HybridClock initialized: now 1790304558321622 us; error 45 us; skew 500 ppm
I20260925 02:49:18.322046 26888 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:18 dist-test-slave-n563 krb5kdc[26875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304558, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:18.325275 26888 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:18.326905 26888 webserver.cc:460] Webserver started at http://127.0.0.1:43955/ using document root <none> and password file <none>
I20260925 02:49:18.327615 26888 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:18.327773 26888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:18.328083 26888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:18.331175 26888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "ee2c7da16e4547cab761f3f35577c47b"
format_stamp: "Formatted at 2026-09-25 02:49:18 on dist-test-slave-n563"
server_key: "e74dcbb35c5b4749487c7db9017d65e2"
server_key_iv: "4cf8bad3e8e4d3bedeb9d3daf46e1dc4"
server_key_version: "encryptionkey@0"
I20260925 02:49:18.331941 26888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "ee2c7da16e4547cab761f3f35577c47b"
format_stamp: "Formatted at 2026-09-25 02:49:18 on dist-test-slave-n563"
server_key: "e74dcbb35c5b4749487c7db9017d65e2"
server_key_iv: "4cf8bad3e8e4d3bedeb9d3daf46e1dc4"
server_key_version: "encryptionkey@0"
I20260925 02:49:18.337275 26888 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:49:18.340792 26903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:18.342099 26888 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:49:18.342280 26888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "ee2c7da16e4547cab761f3f35577c47b"
format_stamp: "Formatted at 2026-09-25 02:49:18 on dist-test-slave-n563"
server_key: "e74dcbb35c5b4749487c7db9017d65e2"
server_key_iv: "4cf8bad3e8e4d3bedeb9d3daf46e1dc4"
server_key_version: "encryptionkey@0"
I20260925 02:49:18.342584 26888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:18.348315 26888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:18.353375 26888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:18.353674 26888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:18.372303 26888 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41003
I20260925 02:49:18.372390 26954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41003 every 8 connection(s)
I20260925 02:49:18.373764 26888 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:18.377688 26955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:18.383270 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 26888
I20260925 02:49:18.383460 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:18.383994 5346 external_mini_cluster.cc:1428] Setting key cd67e19976716d63625657932b574fc8
I20260925 02:49:18.389039 26955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b: Bootstrap starting.
I20260925 02:49:18.392036 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:18.385871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:44413 (local address 127.0.0.1:41003)
0925 02:49:18.386760 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:18.386773 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:18.386820 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:49:18.388536 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:18.388561 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:18.388911 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:18.389306 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:18.390370 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:18.390376 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:18.390448 (+ 72us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:18.390472 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:18.390536 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:18.390556 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:18.390797 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:18.390934 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 02:49:18.391140 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":653,"thread_start_us":405,"threads_started":1}
I20260925 02:49:18.392560 26955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:18.393680 26955 log.cc:824] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:18.395679 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:18.396601 26955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b: No bootstrap required, opened a new log
I20260925 02:49:18.402516 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:18.400105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:37819 (local address 127.0.0.1:41003)
0925 02:49:18.400261 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:18.400266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:18.400587 (+ 321us) server_negotiation.cc:407] Connection header received
0925 02:49:18.400759 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:18.400762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:18.400811 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:18.400896 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:18.401825 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:18.401829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:18.401832 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:18.401847 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:18.401884 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:18.401898 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:18.402010 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:18.402131 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 02:49:18.402176 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 02:49:18.405470 26955 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2c7da16e4547cab761f3f35577c47b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } }
I20260925 02:49:18.405890 26955 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:18.406010 26955 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee2c7da16e4547cab761f3f35577c47b, State: Initialized, Role: FOLLOWER
I20260925 02:49:18.406658 26955 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2c7da16e4547cab761f3f35577c47b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } }
I20260925 02:49:18.406925 26955 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:18.407071 26955 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:18.407250 26955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:18.409732 26955 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2c7da16e4547cab761f3f35577c47b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } }
I20260925 02:49:18.410256 26955 leader_election.cc:302] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee2c7da16e4547cab761f3f35577c47b; no voters:
I20260925 02:49:18.410943 26955 leader_election.cc:288] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:18.411075 26965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:18.412011 26965 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [term 1 LEADER]: Becoming Leader. State: Replica: ee2c7da16e4547cab761f3f35577c47b, State: Running, Role: LEADER
I20260925 02:49:18.412562 26965 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2c7da16e4547cab761f3f35577c47b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } }
I20260925 02:49:18.413102 26955 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:18.415390 26966 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee2c7da16e4547cab761f3f35577c47b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2c7da16e4547cab761f3f35577c47b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } } }
I20260925 02:49:18.415786 26966 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:18.415609 26967 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee2c7da16e4547cab761f3f35577c47b. Latest consensus state: current_term: 1 leader_uuid: "ee2c7da16e4547cab761f3f35577c47b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee2c7da16e4547cab761f3f35577c47b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41003 } } }
I20260925 02:49:18.416005 26967 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:18.419145 26971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:18.424602 26971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:18.433735 26971 catalog_manager.cc:1347] Generated new cluster ID: 04cd7f5bd124403299a30743eb0817e6
I20260925 02:49:18.433892 26971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:18.444304 26971 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:18.444751 26971 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:18.455521 26971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee2c7da16e4547cab761f3f35577c47b: Generated new TSK 0
I20260925 02:49:18.456387 26971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:18.467423 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26888
W20260925 02:49:18.468513 26950 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.130:44413
W20260925 02:49:18.469259 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26888: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:18.469434 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 26888
2026-09-25T02:49:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:18 dist-test-slave-n563 krb5kdc[26986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:18 dist-test-slave-n563 krb5kdc[26986](info): set up 2 sockets
Sep 25 02:49:18 dist-test-slave-n563 krb5kdc[26986](info): commencing operation
krb5kdc: starting...
W20260925 02:49:19.587955 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:19 dist-test-slave-n563 krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:19.861552 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33571
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:20.199436 26999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:20.199743 26999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:20.199962 26999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:20.207409 26999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:20.207556 26999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:20.207646 26999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:20.207757 26999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:20.207847 26999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:20.218122 26999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33571
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:20.219609 26999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:20.221071 26999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:20.233448 27004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:20.233508 27007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:20.233537 27005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:20.234489 26999 server_base.cc:1034] running on GCE node
I20260925 02:49:20.235213 26999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:20.236596 26999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:20.237821 26999 hybrid_clock.cc:648] HybridClock initialized: now 1790304560237777 us; error 27 us; skew 500 ppm
I20260925 02:49:20.238199 26999 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:20 dist-test-slave-n563 krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304560, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:20.241420 26999 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:20.243119 26999 webserver.cc:460] Webserver started at http://127.0.0.1:37743/ using document root <none> and password file <none>
I20260925 02:49:20.243844 26999 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:20.243970 26999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:20.244302 26999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:20.247485 26999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "0428f40d7f6f43c987fb8427d2e5c836"
format_stamp: "Formatted at 2026-09-25 02:49:20 on dist-test-slave-n563"
server_key: "0c30489e7ecc33f6d252cf2a59f2d4d0"
server_key_iv: "b7441293a65a96916b034101ddd63a0f"
server_key_version: "encryptionkey@0"
I20260925 02:49:20.248274 26999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "0428f40d7f6f43c987fb8427d2e5c836"
format_stamp: "Formatted at 2026-09-25 02:49:20 on dist-test-slave-n563"
server_key: "0c30489e7ecc33f6d252cf2a59f2d4d0"
server_key_iv: "b7441293a65a96916b034101ddd63a0f"
server_key_version: "encryptionkey@0"
I20260925 02:49:20.253441 26999 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:49:20.256870 27014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:20.258256 26999 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:49:20.258474 26999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "0428f40d7f6f43c987fb8427d2e5c836"
format_stamp: "Formatted at 2026-09-25 02:49:20 on dist-test-slave-n563"
server_key: "0c30489e7ecc33f6d252cf2a59f2d4d0"
server_key_iv: "b7441293a65a96916b034101ddd63a0f"
server_key_version: "encryptionkey@0"
I20260925 02:49:20.258671 26999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:20.273738 26999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:20.277807 26999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:20.278108 26999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:20.297026 26999 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33571
I20260925 02:49:20.297061 27067 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33571 every 8 connection(s)
I20260925 02:49:20.298498 26999 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:20.302227 27068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:20.308599 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 26999
I20260925 02:49:20.308804 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:20.309355 5346 external_mini_cluster.cc:1428] Setting key 261a62b454e619dcf878e50073d8fefa
I20260925 02:49:20.313750 27068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836: Bootstrap starting.
I20260925 02:49:20.317641 27068 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:20.318926 27068 log.cc:824] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:20.322440 27068 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836: No bootstrap required, opened a new log
Sep 25 02:49:20 dist-test-slave-n563 krb5kdc[26986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 02:49:20.331517 27068 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0428f40d7f6f43c987fb8427d2e5c836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260925 02:49:20.331936 27068 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:20.332070 27068 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0428f40d7f6f43c987fb8427d2e5c836, State: Initialized, Role: FOLLOWER
I20260925 02:49:20.332851 27068 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0428f40d7f6f43c987fb8427d2e5c836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260925 02:49:20.333135 27068 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:20.333280 27068 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:20.333451 27068 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:20.334399 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:20.311350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:34457 (local address 127.0.0.1:33571)
0925 02:49:20.312189 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:20.312200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:20.312242 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:49:20.313967 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:20.313989 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:20.314452 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:20.314792 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:20.316739 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:20.318139 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:20.319946 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:20.320800 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:20.320948 (+ 148us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:20.325978 (+ 5030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:20.325989 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:20.326005 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:20.326047 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:20.329653 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:20.330502 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:20.330509 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:20.330515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:20.330694 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:20.331208 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:20.331212 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:20.331214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:20.331727 (+ 513us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:20.332049 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:20.332590 (+ 541us) server_negotiation.cc:299] Negotiation successful
0925 02:49:20.332945 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":408,"threads_started":1}
I20260925 02:49:20.336208 27068 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0428f40d7f6f43c987fb8427d2e5c836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260925 02:49:20.336715 27068 leader_election.cc:302] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0428f40d7f6f43c987fb8427d2e5c836; no voters:
I20260925 02:49:20.337360 27068 leader_election.cc:288] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:20.337504 27073 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:20.338726 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:20.338654 27073 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [term 1 LEADER]: Becoming Leader. State: Replica: 0428f40d7f6f43c987fb8427d2e5c836, State: Running, Role: LEADER
I20260925 02:49:20.339210 27073 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0428f40d7f6f43c987fb8427d2e5c836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } }
I20260925 02:49:20.339625 27068 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:20.348479 27074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0428f40d7f6f43c987fb8427d2e5c836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0428f40d7f6f43c987fb8427d2e5c836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } } }
I20260925 02:49:20.348989 27074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:20.349608 27075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0428f40d7f6f43c987fb8427d2e5c836. Latest consensus state: current_term: 1 leader_uuid: "0428f40d7f6f43c987fb8427d2e5c836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0428f40d7f6f43c987fb8427d2e5c836" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33571 } } }
I20260925 02:49:20.349938 27075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:20.352236 27087 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:20.362716 27087 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:20.374473 27087 catalog_manager.cc:1347] Generated new cluster ID: 7c3a1b425a9b4622bfaa8956d0614124
I20260925 02:49:20.374591 27087 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:20.380292 27071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:20.353804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:34299 (local address 127.0.0.1:33571)
0925 02:49:20.353979 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:20.353985 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:20.358708 (+ 4723us) server_negotiation.cc:407] Connection header received
0925 02:49:20.359058 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:20.359063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:20.359159 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:20.359293 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:20.361335 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:20.362630 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:20.364642 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:20.366553 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:20.366663 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:20.373080 (+ 6417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:20.373087 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:20.373091 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:20.373126 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:20.376880 (+ 3754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:20.377847 (+ 967us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:20.377853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:20.377856 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:20.378007 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:20.378536 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:20.378541 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:20.378544 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:20.378933 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:20.379162 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:20.379777 (+ 615us) server_negotiation.cc:299] Negotiation successful
0925 02:49:20.379955 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260925 02:49:20.394410 27087 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:20.396319 27087 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:20.405019 27087 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0428f40d7f6f43c987fb8427d2e5c836: Generated new TSK 0
I20260925 02:49:20.406183 27087 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:20.415737 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 26999
I20260925 02:49:20.416955 27063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:20.510003 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 26999
W20260925 02:49:20.529563 26984 connection.cc:535] client connection to 127.0.0.1:33571 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33571): Connection reset by peer (error 104)
2026-09-25T02:49:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2042 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:20 dist-test-slave-n563 krb5kdc[27101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:20 dist-test-slave-n563 krb5kdc[27101](info): set up 2 sockets
Sep 25 02:49:20 dist-test-slave-n563 krb5kdc[27101](info): commencing operation
krb5kdc: starting...
W20260925 02:49:21.628135 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:21 dist-test-slave-n563 krb5kdc[27101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:21.901930 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:22.239916 27114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:22.240306 27114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:22.240473 27114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:22.247900 27114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:22.248080 27114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:22.248198 27114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:22.248318 27114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:22.248406 27114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:22.258484 27114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39463
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:22.259863 27114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:22.261228 27114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:22.273370 27119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:22.273370 27122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:22.273411 27120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:22.274416 27114 server_base.cc:1034] running on GCE node
I20260925 02:49:22.275220 27114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:22.276619 27114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:22.277824 27114 hybrid_clock.cc:648] HybridClock initialized: now 1790304562277781 us; error 31 us; skew 500 ppm
I20260925 02:49:22.278189 27114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:22 dist-test-slave-n563 krb5kdc[27101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304562, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:22.286495 27114 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:22.288113 27114 webserver.cc:460] Webserver started at http://127.0.0.1:35921/ using document root <none> and password file <none>
I20260925 02:49:22.288782 27114 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:22.288884 27114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:22.289142 27114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:22.292234 27114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "1cc1ec2763b9477bb1d3bc86d4412d46"
format_stamp: "Formatted at 2026-09-25 02:49:22 on dist-test-slave-n563"
server_key: "42591abbdc2bb8eeef0bace8a5b9b7c4"
server_key_iv: "d85b9fdede7c873b00b7173ee5998801"
server_key_version: "encryptionkey@0"
I20260925 02:49:22.292991 27114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "1cc1ec2763b9477bb1d3bc86d4412d46"
format_stamp: "Formatted at 2026-09-25 02:49:22 on dist-test-slave-n563"
server_key: "42591abbdc2bb8eeef0bace8a5b9b7c4"
server_key_iv: "d85b9fdede7c873b00b7173ee5998801"
server_key_version: "encryptionkey@0"
I20260925 02:49:22.298594 27114 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260925 02:49:22.302067 27129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:22.303570 27114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:49:22.303767 27114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "1cc1ec2763b9477bb1d3bc86d4412d46"
format_stamp: "Formatted at 2026-09-25 02:49:22 on dist-test-slave-n563"
server_key: "42591abbdc2bb8eeef0bace8a5b9b7c4"
server_key_iv: "d85b9fdede7c873b00b7173ee5998801"
server_key_version: "encryptionkey@0"
I20260925 02:49:22.303978 27114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:22.333756 27114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:22.338445 27114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:22.338733 27114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:22.357650 27114 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39463
I20260925 02:49:22.357692 27181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39463 every 8 connection(s)
I20260925 02:49:22.359082 27114 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:22.362867 27182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:22.369230 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 27114
I20260925 02:49:22.369458 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:22.369925 5346 external_mini_cluster.cc:1428] Setting key 68733091f60192c4c52186c28f939dee
I20260925 02:49:22.374143 27182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46: Bootstrap starting.
I20260925 02:49:22.377782 27182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:22.378979 27182 log.cc:824] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:22.382516 27182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46: No bootstrap required, opened a new log
I20260925 02:49:22.384434 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:22.371919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:55675 (local address 127.0.0.1:39463)
0925 02:49:22.372723 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:22.372735 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:22.372777 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:49:22.374536 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:22.374559 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:22.374980 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:22.375326 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:22.375974 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:22.377263 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:22.379071 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:22.380155 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:22.380294 (+ 139us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:22.381556 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:22.381567 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:22.381587 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:22.382303 (+ 716us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:22.382430 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:22.382463 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:22.382647 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:22.382923 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 02:49:22.383127 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":589,"thread_start_us":384,"threads_started":1}
I20260925 02:49:22.388492 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:22.391700 27182 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1ec2763b9477bb1d3bc86d4412d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39463 } }
I20260925 02:49:22.392084 27182 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:22.392184 27182 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc1ec2763b9477bb1d3bc86d4412d46, State: Initialized, Role: FOLLOWER
I20260925 02:49:22.392874 27182 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1ec2763b9477bb1d3bc86d4412d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39463 } }
I20260925 02:49:22.393213 27182 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:22.393354 27182 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:22.393550 27182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:22.396201 27182 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1ec2763b9477bb1d3bc86d4412d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39463 } }
I20260925 02:49:22.396709 27182 leader_election.cc:302] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc1ec2763b9477bb1d3bc86d4412d46; no voters:
I20260925 02:49:22.397545 27182 leader_election.cc:288] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:22.397697 27192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:22.400285 27182 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:22.400328 27185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:22.393124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:43833 (local address 127.0.0.1:39463)
0925 02:49:22.393951 (+ 827us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:22.393957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:22.394006 (+ 49us) server_negotiation.cc:407] Connection header received
0925 02:49:22.394093 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:22.394099 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:22.394205 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:22.394338 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:22.394964 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:22.395744 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:22.397205 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:22.397872 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:22.398035 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:22.399482 (+ 1447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:22.399490 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:22.399500 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:22.399517 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:22.399576 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:22.399593 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:22.399715 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:22.399933 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 02:49:22.399988 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 02:49:22.402720 27192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc1ec2763b9477bb1d3bc86d4412d46, State: Running, Role: LEADER
I20260925 02:49:22.403468 27192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1ec2763b9477bb1d3bc86d4412d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39463 } }
I20260925 02:49:22.406646 27194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc1ec2763b9477bb1d3bc86d4412d46. Latest consensus state: current_term: 1 leader_uuid: "1cc1ec2763b9477bb1d3bc86d4412d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1ec2763b9477bb1d3bc86d4412d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39463 } } }
I20260925 02:49:22.407702 27194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:22.407169 27193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cc1ec2763b9477bb1d3bc86d4412d46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc1ec2763b9477bb1d3bc86d4412d46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39463 } } }
I20260925 02:49:22.408130 27193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46 [sys.catalog]: This master's current role is: LEADER
W20260925 02:49:22.411906 27206 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:49:22.412024 27206 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:49:22.412195 27207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:22.414497 27207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:22.422695 27207 catalog_manager.cc:1347] Generated new cluster ID: e7825403ced242788b98e6708e1e3386
I20260925 02:49:22.422814 27207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:22.444669 27207 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:22.446010 27207 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:22.455799 27207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cc1ec2763b9477bb1d3bc86d4412d46: Generated new TSK 0
I20260925 02:49:22.456988 27207 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:22.476096 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27114
W20260925 02:49:22.477181 27177 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.130:55675
W20260925 02:49:22.477975 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27114: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:22.478142 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 27114
W20260925 02:49:22.496131 27099 connection.cc:535] client connection to 127.0.0.1:39463 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39463): Connection reset by peer (error 104)
2026-09-25T02:49:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1966 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:22 dist-test-slave-n563 krb5kdc[27214](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:22 dist-test-slave-n563 krb5kdc[27214](info): set up 2 sockets
Sep 25 02:49:22 dist-test-slave-n563 krb5kdc[27214](info): commencing operation
krb5kdc: starting...
W20260925 02:49:23.629496 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:23 dist-test-slave-n563 krb5kdc[27214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304563, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 02:49:23.899761 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44839
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44839
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:24.242179 27227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:24.242583 27227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:24.242689 27227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:24.250219 27227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:24.250351 27227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:24.250463 27227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:24.250550 27227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:24.250592 27227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:24.261162 27227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44839
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44839
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:24.262528 27227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:24.263897 27227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:24.276810 27235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:24.276980 27232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:24.276947 27233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:24.278079 27227 server_base.cc:1034] running on GCE node
I20260925 02:49:24.279032 27227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:24.280548 27227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:24.281772 27227 hybrid_clock.cc:648] HybridClock initialized: now 1790304564281717 us; error 42 us; skew 500 ppm
I20260925 02:49:24.282151 27227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:24 dist-test-slave-n563 krb5kdc[27214](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304564, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:24.285475 27227 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 02:49:24.287158 27227 webserver.cc:460] Webserver started at http://127.0.0.1:37171/ using document root <none> and password file <none>
I20260925 02:49:24.287884 27227 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:24.288038 27227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:24.288359 27227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:24.291618 27227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "88a60481b5e3435b966453dd44cc9e78"
format_stamp: "Formatted at 2026-09-25 02:49:24 on dist-test-slave-n563"
server_key: "e791ac6875f8a17878e6f6a7dcce56a7"
server_key_iv: "9fbd4e478b92ab3b1236ec3507fa0746"
server_key_version: "encryptionkey@0"
I20260925 02:49:24.292434 27227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "88a60481b5e3435b966453dd44cc9e78"
format_stamp: "Formatted at 2026-09-25 02:49:24 on dist-test-slave-n563"
server_key: "e791ac6875f8a17878e6f6a7dcce56a7"
server_key_iv: "9fbd4e478b92ab3b1236ec3507fa0746"
server_key_version: "encryptionkey@0"
I20260925 02:49:24.298233 27227 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:49:24.301977 27242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:24.303439 27227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:49:24.303640 27227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "88a60481b5e3435b966453dd44cc9e78"
format_stamp: "Formatted at 2026-09-25 02:49:24 on dist-test-slave-n563"
server_key: "e791ac6875f8a17878e6f6a7dcce56a7"
server_key_iv: "9fbd4e478b92ab3b1236ec3507fa0746"
server_key_version: "encryptionkey@0"
I20260925 02:49:24.303875 27227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:24.310256 27227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:24.314416 27227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:24.314769 27227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:24.334250 27227 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44839
I20260925 02:49:24.334291 27293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44839 every 8 connection(s)
I20260925 02:49:24.335845 27227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:24.336485 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 27227
I20260925 02:49:24.336688 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:24.337287 5346 external_mini_cluster.cc:1428] Setting key cdbb86425fd28b5252ccdc8df6e47c8d
I20260925 02:49:24.340444 27294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:24.345639 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:24.339201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:60105 (local address 127.0.0.1:44839)
0925 02:49:24.340092 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:24.340104 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:24.340145 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:49:24.342019 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:24.342044 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:24.342479 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:24.342824 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:24.343958 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:24.343964 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:24.343976 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:24.343997 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:24.344066 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:24.344089 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:24.344256 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:24.344416 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 02:49:24.344644 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":652,"thread_start_us":438,"threads_started":1}
I20260925 02:49:24.349645 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:24.353082 27294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78: Bootstrap starting.
I20260925 02:49:24.356565 27294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:24.356858 27297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:24.354069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.130:38977 (local address 127.0.0.1:44839)
0925 02:49:24.354228 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:24.354232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:24.354805 (+ 573us) server_negotiation.cc:407] Connection header received
0925 02:49:24.355008 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:24.355013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:24.355081 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:24.355183 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:24.356133 (+ 950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:24.356139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:24.356144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:24.356162 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:24.356213 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:24.356228 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:24.356347 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:24.356486 (+ 139us) server_negotiation.cc:299] Negotiation successful
0925 02:49:24.356557 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 02:49:24.357854 27294 log.cc:824] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:24.360688 27294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78: No bootstrap required, opened a new log
I20260925 02:49:24.368451 27294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a60481b5e3435b966453dd44cc9e78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44839 } }
I20260925 02:49:24.368934 27294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:24.369045 27294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88a60481b5e3435b966453dd44cc9e78, State: Initialized, Role: FOLLOWER
I20260925 02:49:24.369867 27294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a60481b5e3435b966453dd44cc9e78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44839 } }
I20260925 02:49:24.370224 27304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 02:49:24.370594 27304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a60481b5e3435b966453dd44cc9e78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44839 } }
I20260925 02:49:24.371311 27304 leader_election.cc:302] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88a60481b5e3435b966453dd44cc9e78; no voters:
I20260925 02:49:24.370152 27294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:24.372089 27294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:24.372205 27304 leader_election.cc:288] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260925 02:49:24.372241 27294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:24.374585 27294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a60481b5e3435b966453dd44cc9e78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44839 } }
I20260925 02:49:24.374886 27294 leader_election.cc:302] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88a60481b5e3435b966453dd44cc9e78; no voters:
I20260925 02:49:24.375000 27305 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260925 02:49:24.375136 27294 leader_election.cc:288] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:24.375532 27304 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:24.376443 27304 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [term 1 LEADER]: Becoming Leader. State: Replica: 88a60481b5e3435b966453dd44cc9e78, State: Running, Role: LEADER
I20260925 02:49:24.376981 27304 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a60481b5e3435b966453dd44cc9e78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44839 } }
I20260925 02:49:24.377377 27294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:24.382714 27305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88a60481b5e3435b966453dd44cc9e78. Latest consensus state: current_term: 1 leader_uuid: "88a60481b5e3435b966453dd44cc9e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a60481b5e3435b966453dd44cc9e78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44839 } } }
I20260925 02:49:24.382948 27306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88a60481b5e3435b966453dd44cc9e78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88a60481b5e3435b966453dd44cc9e78" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44839 } } }
I20260925 02:49:24.383457 27306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:24.383488 27305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78 [sys.catalog]: This master's current role is: LEADER
W20260925 02:49:24.399122 27320 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:49:24.399269 27320 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:49:24.399390 27313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:24.401050 27313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:24.410063 27313 catalog_manager.cc:1347] Generated new cluster ID: 1a5342322c5c4014a1f5c729525fbbef
I20260925 02:49:24.410233 27313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:24.482614 27313 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:24.483116 27313 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:24.492336 27313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88a60481b5e3435b966453dd44cc9e78: Generated new TSK 0
I20260925 02:49:24.493271 27313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:24.500612 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27227
W20260925 02:49:24.502455 27289 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.130:60105
W20260925 02:49:24.503228 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27227: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:24.503389 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 27227
2026-09-25T02:49:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2025 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:24 dist-test-slave-n563 krb5kdc[27326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:24 dist-test-slave-n563 krb5kdc[27326](info): set up 2 sockets
Sep 25 02:49:24 dist-test-slave-n563 krb5kdc[27326](info): commencing operation
krb5kdc: starting...
W20260925 02:49:25.628026 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.009s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:25 dist-test-slave-n563 krb5kdc[27326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:49:25.913554 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43703
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:43703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:26.248175 27339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:26.248476 27339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:26.248648 27339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:26.255898 27339 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:26.256009 27339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:26.256104 27339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:26.256211 27339 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:26.256274 27339 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:26.266141 27339 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:43703
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:43703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:26.267552 27339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:26.268872 27339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:26.281008 27345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:26.281148 27348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:26.281013 27344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:26.281701 27339 server_base.cc:1034] running on GCE node
I20260925 02:49:26.282667 27339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:26.284183 27339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:26.285421 27339 hybrid_clock.cc:648] HybridClock initialized: now 1790304566285372 us; error 42 us; skew 500 ppm
I20260925 02:49:26.285787 27339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:26 dist-test-slave-n563 krb5kdc[27326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304566, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:26.288985 27339 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:49:26.290557 27339 webserver.cc:460] Webserver started at http://127.5.56.190:37145/ using document root <none> and password file <none>
I20260925 02:49:26.291240 27339 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:26.291374 27339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:26.291704 27339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:26.294857 27339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "7b3adc2397894630bcfc2a191643e4b7"
format_stamp: "Formatted at 2026-09-25 02:49:26 on dist-test-slave-n563"
server_key: "22306eee18f94d78bcd66f8c54bfb926"
server_key_iv: "110cb454bd93c688938671696b3bde1e"
server_key_version: "encryptionkey@0"
I20260925 02:49:26.295614 27339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "7b3adc2397894630bcfc2a191643e4b7"
format_stamp: "Formatted at 2026-09-25 02:49:26 on dist-test-slave-n563"
server_key: "22306eee18f94d78bcd66f8c54bfb926"
server_key_iv: "110cb454bd93c688938671696b3bde1e"
server_key_version: "encryptionkey@0"
I20260925 02:49:26.301440 27339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260925 02:49:26.305059 27355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:26.306442 27339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:49:26.306643 27339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "7b3adc2397894630bcfc2a191643e4b7"
format_stamp: "Formatted at 2026-09-25 02:49:26 on dist-test-slave-n563"
server_key: "22306eee18f94d78bcd66f8c54bfb926"
server_key_iv: "110cb454bd93c688938671696b3bde1e"
server_key_version: "encryptionkey@0"
I20260925 02:49:26.306857 27339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:26.332433 27339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:26.336639 27339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:26.336946 27339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:26.355257 27339 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:43703
I20260925 02:49:26.355289 27406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:43703 every 8 connection(s)
I20260925 02:49:26.356768 27339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:26.360209 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 27339
I20260925 02:49:26.360430 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:26.360808 27407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:26.360908 5346 external_mini_cluster.cc:1428] Setting key 081a44c432d3675296fc45a67e95930c
I20260925 02:49:26.375803 27407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7: Bootstrap starting.
Sep 25 02:49:26 dist-test-slave-n563 krb5kdc[27326](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:49:26.380043 27407 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:26.381482 27407 log.cc:824] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:26.384478 27407 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7: No bootstrap required, opened a new log
I20260925 02:49:26.385869 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:26.362868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57983 (local address 127.5.56.190:43703)
0925 02:49:26.363688 (+ 820us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:26.363699 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:26.363743 (+ 44us) server_negotiation.cc:407] Connection header received
0925 02:49:26.365652 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:26.365675 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:26.366104 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:26.366509 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:26.368257 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:26.369538 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:26.371445 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:26.372149 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:26.372290 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:26.377274 (+ 4984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:26.377289 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:26.377315 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:26.377370 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:26.381434 (+ 4064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:26.382420 (+ 986us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:26.382432 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:26.382446 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:26.382623 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:26.383107 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:26.383111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:26.383113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:26.383647 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:26.383989 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:26.384500 (+ 511us) server_negotiation.cc:299] Negotiation successful
0925 02:49:26.384858 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":374,"threads_started":1}
I20260925 02:49:26.390106 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:26.393450 27407 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3adc2397894630bcfc2a191643e4b7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43703 } }
I20260925 02:49:26.393882 27407 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:26.393993 27407 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b3adc2397894630bcfc2a191643e4b7, State: Initialized, Role: FOLLOWER
I20260925 02:49:26.394788 27407 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3adc2397894630bcfc2a191643e4b7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43703 } }
I20260925 02:49:26.395108 27407 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:26.395260 27407 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:26.395478 27407 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:26.398689 27407 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3adc2397894630bcfc2a191643e4b7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43703 } }
I20260925 02:49:26.399317 27407 leader_election.cc:302] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b3adc2397894630bcfc2a191643e4b7; no voters:
I20260925 02:49:26.400002 27407 leader_election.cc:288] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:26.400131 27417 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:26.402124 27407 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:26.402695 27417 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [term 1 LEADER]: Becoming Leader. State: Replica: 7b3adc2397894630bcfc2a191643e4b7, State: Running, Role: LEADER
I20260925 02:49:26.403302 27417 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3adc2397894630bcfc2a191643e4b7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43703 } }
I20260925 02:49:26.405967 27419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b3adc2397894630bcfc2a191643e4b7. Latest consensus state: current_term: 1 leader_uuid: "7b3adc2397894630bcfc2a191643e4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3adc2397894630bcfc2a191643e4b7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43703 } } }
I20260925 02:49:26.406414 27419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:26.407718 27418 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b3adc2397894630bcfc2a191643e4b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b3adc2397894630bcfc2a191643e4b7" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 43703 } } }
I20260925 02:49:26.409243 27418 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:26.410553 27427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:26.416244 27427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:26.416953 27410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:26.394690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51515 (local address 127.5.56.190:43703)
0925 02:49:26.394938 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:26.394948 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:26.395353 (+ 405us) server_negotiation.cc:407] Connection header received
0925 02:49:26.395598 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:26.395605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:26.395688 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:26.395770 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:26.397675 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:26.398513 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:26.400287 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:26.401181 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:26.401341 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:26.404919 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:26.404927 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:26.404938 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:26.404981 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:26.411525 (+ 6544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:26.412624 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:26.412632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:26.412638 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:26.412848 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:26.413622 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:26.413629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:26.413635 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:26.414225 (+ 590us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:26.415879 (+ 1654us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:26.416136 (+ 257us) server_negotiation.cc:299] Negotiation successful
0925 02:49:26.416396 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:49:26.426244 27427 catalog_manager.cc:1347] Generated new cluster ID: 2bae07d397d44759aeb7784feaca2bc9
I20260925 02:49:26.426367 27427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:26.446776 27427 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:26.448120 27427 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:26.455879 27427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b3adc2397894630bcfc2a191643e4b7: Generated new TSK 0
I20260925 02:49:26.456734 27427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:26.463557 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27339
I20260925 02:49:26.464730 27402 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:26.556936 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 27339
W20260925 02:49:26.576529 27324 connection.cc:535] client connection to 127.5.56.190:43703 recv error: Network error: failed to read from TLS socket (remote: 127.5.56.190:43703): Connection reset by peer (error 104)
2026-09-25T02:49:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2053 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:26 dist-test-slave-n563 krb5kdc[27439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:26 dist-test-slave-n563 krb5kdc[27439](info): set up 2 sockets
Sep 25 02:49:26 dist-test-slave-n563 krb5kdc[27439](info): commencing operation
krb5kdc: starting...
W20260925 02:49:27.676051 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.081s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:27 dist-test-slave-n563 krb5kdc[27439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304567, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:49:27.946367 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:33829
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:35145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:33829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:28.283704 27452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:28.284034 27452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:28.284138 27452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:28.291409 27452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:28.291534 27452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:28.291582 27452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:28.291646 27452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:28.291690 27452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:28.301855 27452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:35145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:33829
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:33829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:28.303258 27452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:28.304531 27452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:28.316753 27457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:28.316802 27458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:28.317291 27460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:28.317617 27452 server_base.cc:1034] running on GCE node
I20260925 02:49:28.318490 27452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:28.319949 27452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:28.321159 27452 hybrid_clock.cc:648] HybridClock initialized: now 1790304568321113 us; error 44 us; skew 500 ppm
I20260925 02:49:28.321532 27452 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:28 dist-test-slave-n563 krb5kdc[27439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304568, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:28.330863 27452 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:49:28.332507 27452 webserver.cc:460] Webserver started at http://127.5.56.190:33973/ using document root <none> and password file <none>
I20260925 02:49:28.333233 27452 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:28.333386 27452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:28.333761 27452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:28.336934 27452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "f70be516c44f415e8cfcddce89bf70ce"
format_stamp: "Formatted at 2026-09-25 02:49:28 on dist-test-slave-n563"
server_key: "c68fa198bd2dd3a753870d017776272a"
server_key_iv: "1b38c6184709b634f2520ae57440b839"
server_key_version: "encryptionkey@0"
I20260925 02:49:28.337724 27452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "f70be516c44f415e8cfcddce89bf70ce"
format_stamp: "Formatted at 2026-09-25 02:49:28 on dist-test-slave-n563"
server_key: "c68fa198bd2dd3a753870d017776272a"
server_key_iv: "1b38c6184709b634f2520ae57440b839"
server_key_version: "encryptionkey@0"
I20260925 02:49:28.343237 27452 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:49:28.346745 27467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:28.348112 27452 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:49:28.348306 27452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "f70be516c44f415e8cfcddce89bf70ce"
format_stamp: "Formatted at 2026-09-25 02:49:28 on dist-test-slave-n563"
server_key: "c68fa198bd2dd3a753870d017776272a"
server_key_iv: "1b38c6184709b634f2520ae57440b839"
server_key_version: "encryptionkey@0"
I20260925 02:49:28.348518 27452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:28.371104 27452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:28.375891 27452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:28.376214 27452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:28.395632 27452 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:33829
I20260925 02:49:28.395654 27519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:33829 every 8 connection(s)
I20260925 02:49:28.397049 27452 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:28.400815 27520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:28.404289 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 27452
I20260925 02:49:28.404462 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:28.404917 5346 external_mini_cluster.cc:1428] Setting key eca58bb29707f98d79ad272b5d5c0d00
I20260925 02:49:28.414608 27520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce: Bootstrap starting.
I20260925 02:49:28.418689 27520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:28.420143 27520 log.cc:824] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:28.420521 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:28.406868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56919 (local address 127.5.56.190:33829)
0925 02:49:28.407741 (+ 873us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:28.407756 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:28.407809 (+ 53us) server_negotiation.cc:407] Connection header received
0925 02:49:28.409916 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:28.409948 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:28.410547 (+ 599us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:28.410999 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:28.411562 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:28.413634 (+ 2072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:28.415082 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:28.416125 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:28.416327 (+ 202us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:28.417553 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:28.417564 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:28.417581 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:28.418324 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:28.418455 (+ 131us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:28.418494 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:28.418683 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:28.418928 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 02:49:28.419134 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":349,"threads_started":1}
I20260925 02:49:28.424451 27520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce: No bootstrap required, opened a new log
I20260925 02:49:28.424744 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:28.435128 27520 raft_consensus.cc:348] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70be516c44f415e8cfcddce89bf70ce" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33829 } }
I20260925 02:49:28.435564 27520 raft_consensus.cc:374] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:28.435673 27520 raft_consensus.cc:729] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f70be516c44f415e8cfcddce89bf70ce, State: Initialized, Role: FOLLOWER
I20260925 02:49:28.436338 27520 consensus_queue.cc:260] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70be516c44f415e8cfcddce89bf70ce" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33829 } }
I20260925 02:49:28.436718 27520 raft_consensus.cc:388] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:28.436918 27520 raft_consensus.cc:482] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:28.436954 27523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:28.430082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55123 (local address 127.5.56.190:33829)
0925 02:49:28.430277 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:28.430284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:28.430637 (+ 353us) server_negotiation.cc:407] Connection header received
0925 02:49:28.430802 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:28.430809 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:28.430893 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:28.431007 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:28.431506 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:28.432687 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:28.434003 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:28.434661 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:28.434795 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:28.435951 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:28.435955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:28.435977 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:28.436004 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:28.436057 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:28.436087 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:28.436196 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:28.436435 (+ 239us) server_negotiation.cc:299] Negotiation successful
0925 02:49:28.436480 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:49:28.437148 27520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:28.439703 27520 raft_consensus.cc:504] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70be516c44f415e8cfcddce89bf70ce" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33829 } }
I20260925 02:49:28.440239 27520 leader_election.cc:302] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f70be516c44f415e8cfcddce89bf70ce; no voters:
I20260925 02:49:28.440873 27520 leader_election.cc:288] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:28.440964 27530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:28.441812 27530 raft_consensus.cc:686] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [term 1 LEADER]: Becoming Leader. State: Replica: f70be516c44f415e8cfcddce89bf70ce, State: Running, Role: LEADER
I20260925 02:49:28.442524 27530 consensus_queue.cc:237] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70be516c44f415e8cfcddce89bf70ce" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33829 } }
I20260925 02:49:28.443349 27520 sys_catalog.cc:564] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:28.445438 27532 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [sys.catalog]: SysCatalogTable state changed. Reason: New leader f70be516c44f415e8cfcddce89bf70ce. Latest consensus state: current_term: 1 leader_uuid: "f70be516c44f415e8cfcddce89bf70ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70be516c44f415e8cfcddce89bf70ce" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33829 } } }
I20260925 02:49:28.445462 27531 sys_catalog.cc:455] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f70be516c44f415e8cfcddce89bf70ce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f70be516c44f415e8cfcddce89bf70ce" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 33829 } } }
I20260925 02:49:28.445907 27532 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:28.445953 27531 sys_catalog.cc:458] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:28.451220 27535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:28.459416 27535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:28.467871 27535 catalog_manager.cc:1347] Generated new cluster ID: 5b67fb319f22476c8763b234f44c3e37
I20260925 02:49:28.468015 27535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:28.481343 27535 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:28.483209 27535 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:28.497325 27535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f70be516c44f415e8cfcddce89bf70ce: Generated new TSK 0
I20260925 02:49:28.498584 27535 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:28.515472 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27452
W20260925 02:49:28.516633 27515 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56919
W20260925 02:49:28.517374 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27452: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:28.517519 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 27452
W20260925 02:49:28.535379 27437 connection.cc:535] client connection to 127.5.56.190:33829 recv error: Network error: failed to read from TLS socket (remote: 127.5.56.190:33829): Connection reset by peer (error 104)
2026-09-25T02:49:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1959 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:28 dist-test-slave-n563 krb5kdc[27551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:28 dist-test-slave-n563 krb5kdc[27551](info): set up 2 sockets
Sep 25 02:49:28 dist-test-slave-n563 krb5kdc[27551](info): commencing operation
krb5kdc: starting...
W20260925 02:49:29.952481 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.398s user 0.000s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:30 dist-test-slave-n563 krb5kdc[27551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:49:30.229234 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:42387
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:42387
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:30.571249 27564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:30.571599 27564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:30.571777 27564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:30.579224 27564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:30.579380 27564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:30.579487 27564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:30.579598 27564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:30.579684 27564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:30.589884 27564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:42387
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:42387
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:30.591317 27564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:30.592700 27564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:30.604823 27569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:30.604748 27570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:30.605050 27572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:30.605708 27564 server_base.cc:1034] running on GCE node
I20260925 02:49:30.606639 27564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:30.608127 27564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:30.609390 27564 hybrid_clock.cc:648] HybridClock initialized: now 1790304570609342 us; error 52 us; skew 500 ppm
I20260925 02:49:30.609759 27564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:30 dist-test-slave-n563 krb5kdc[27551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304570, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:30.612927 27564 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:49:30.614477 27564 webserver.cc:460] Webserver started at http://127.5.56.190:34457/ using document root <none> and password file <none>
I20260925 02:49:30.615289 27564 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:30.615439 27564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:30.615752 27564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:30.618914 27564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "9a619cbce7d14a0bbe3d513d334fca85"
format_stamp: "Formatted at 2026-09-25 02:49:30 on dist-test-slave-n563"
server_key: "3e93f68aac6faf4bd4664f3a625eaf2f"
server_key_iv: "331e38ea54adcc1662a6b4bd18d2d24b"
server_key_version: "encryptionkey@0"
I20260925 02:49:30.619730 27564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "9a619cbce7d14a0bbe3d513d334fca85"
format_stamp: "Formatted at 2026-09-25 02:49:30 on dist-test-slave-n563"
server_key: "3e93f68aac6faf4bd4664f3a625eaf2f"
server_key_iv: "331e38ea54adcc1662a6b4bd18d2d24b"
server_key_version: "encryptionkey@0"
I20260925 02:49:30.625142 27564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260925 02:49:30.628777 27579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:30.630122 27564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260925 02:49:30.630307 27564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "9a619cbce7d14a0bbe3d513d334fca85"
format_stamp: "Formatted at 2026-09-25 02:49:30 on dist-test-slave-n563"
server_key: "3e93f68aac6faf4bd4664f3a625eaf2f"
server_key_iv: "331e38ea54adcc1662a6b4bd18d2d24b"
server_key_version: "encryptionkey@0"
I20260925 02:49:30.630576 27564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:30.636412 27564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:30.640413 27564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:30.640719 27564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:30.658928 27564 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:42387
I20260925 02:49:30.658950 27630 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:42387 every 8 connection(s)
I20260925 02:49:30.660392 27564 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:30.664088 27631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:30.666378 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 27564
I20260925 02:49:30.666625 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:30.667096 5346 external_mini_cluster.cc:1428] Setting key 14b9dca086458561fe4c651048748505
I20260925 02:49:30.674863 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:30.668885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60305 (local address 127.5.56.190:42387)
0925 02:49:30.669736 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:30.669750 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:30.669789 (+ 39us) server_negotiation.cc:407] Connection header received
0925 02:49:30.671415 (+ 1626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:30.671437 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:30.671799 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:30.672198 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:30.673218 (+ 1020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:30.673224 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:30.673237 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:30.673257 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:30.673309 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:30.673328 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:30.673576 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:30.673705 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 02:49:30.673890 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":425,"threads_started":1}
I20260925 02:49:30.678910 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:30.679096 27631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85: Bootstrap starting.
I20260925 02:49:30.682340 27631 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:30.683656 27631 log.cc:824] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:30.685928 27634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:30.683412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33095 (local address 127.5.56.190:42387)
0925 02:49:30.683602 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:30.683608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:30.683951 (+ 343us) server_negotiation.cc:407] Connection header received
0925 02:49:30.684098 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:30.684105 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:30.684194 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:30.684344 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:30.685280 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:30.685285 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:30.685290 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:30.685309 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:30.685356 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:30.685378 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:30.685507 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:30.685562 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 02:49:30.685623 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:49:30.687255 27631 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85: No bootstrap required, opened a new log
I20260925 02:49:30.695390 27631 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a619cbce7d14a0bbe3d513d334fca85" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42387 } }
I20260925 02:49:30.695842 27631 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:30.695987 27631 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a619cbce7d14a0bbe3d513d334fca85, State: Initialized, Role: FOLLOWER
I20260925 02:49:30.696645 27631 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a619cbce7d14a0bbe3d513d334fca85" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42387 } }
I20260925 02:49:30.696914 27631 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:30.697103 27631 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:30.697306 27631 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:30.699946 27631 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a619cbce7d14a0bbe3d513d334fca85" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42387 } }
I20260925 02:49:30.700496 27631 leader_election.cc:302] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a619cbce7d14a0bbe3d513d334fca85; no voters:
I20260925 02:49:30.701176 27631 leader_election.cc:288] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:30.701331 27642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:30.702715 27642 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [term 1 LEADER]: Becoming Leader. State: Replica: 9a619cbce7d14a0bbe3d513d334fca85, State: Running, Role: LEADER
I20260925 02:49:30.703248 27642 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a619cbce7d14a0bbe3d513d334fca85" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42387 } }
I20260925 02:49:30.703778 27631 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:30.706099 27643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a619cbce7d14a0bbe3d513d334fca85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a619cbce7d14a0bbe3d513d334fca85" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42387 } } }
I20260925 02:49:30.706069 27644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a619cbce7d14a0bbe3d513d334fca85. Latest consensus state: current_term: 1 leader_uuid: "9a619cbce7d14a0bbe3d513d334fca85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a619cbce7d14a0bbe3d513d334fca85" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 42387 } } }
I20260925 02:49:30.706562 27643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:30.706575 27644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:30.708350 27648 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:30.714011 27648 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:30.723909 27648 catalog_manager.cc:1347] Generated new cluster ID: a197f68da07d425cbee51528bf51969d
I20260925 02:49:30.724046 27648 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:30.742077 27648 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:30.742594 27648 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:30.752815 27648 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a619cbce7d14a0bbe3d513d334fca85: Generated new TSK 0
I20260925 02:49:30.753713 27648 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:30.764952 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27564
W20260925 02:49:30.766172 27626 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60305
W20260925 02:49:30.767016 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27564: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:30.767175 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 27564
2026-09-25T02:49:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2249 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:30 dist-test-slave-n563 krb5kdc[27663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:30 dist-test-slave-n563 krb5kdc[27663](info): set up 2 sockets
Sep 25 02:49:30 dist-test-slave-n563 krb5kdc[27663](info): commencing operation
krb5kdc: starting...
W20260925 02:49:31.928247 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.124s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:32 dist-test-slave-n563 krb5kdc[27663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:49:32.197621 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41875
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:45665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:41875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:32.533425 27676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:32.533727 27676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:32.533879 27676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:32.541229 27676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:32.541363 27676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:32.541491 27676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:32.541610 27676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:32.541692 27676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:32.551864 27676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:45665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:41875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:41875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:32.553237 27676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:32.554612 27676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:32.566727 27682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:32.566776 27684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:32.567008 27676 server_base.cc:1034] running on GCE node
W20260925 02:49:32.566793 27681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:32.568120 27676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:32.569583 27676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:32.570813 27676 hybrid_clock.cc:648] HybridClock initialized: now 1790304572570762 us; error 66 us; skew 500 ppm
I20260925 02:49:32.571172 27676 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:32 dist-test-slave-n563 krb5kdc[27663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304572, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:32.574321 27676 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:49:32.576040 27676 webserver.cc:460] Webserver started at http://127.5.56.190:42035/ using document root <none> and password file <none>
I20260925 02:49:32.576965 27676 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:32.577108 27676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:32.577476 27676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:32.580858 27676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "ac1c2bfa50a443cabcb393b10c547761"
format_stamp: "Formatted at 2026-09-25 02:49:32 on dist-test-slave-n563"
server_key: "203f9aad80b354002acb162e1cf0d7fc"
server_key_iv: "348ba0e331d81830b9bff12a3359791c"
server_key_version: "encryptionkey@0"
I20260925 02:49:32.581571 27676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "ac1c2bfa50a443cabcb393b10c547761"
format_stamp: "Formatted at 2026-09-25 02:49:32 on dist-test-slave-n563"
server_key: "203f9aad80b354002acb162e1cf0d7fc"
server_key_iv: "348ba0e331d81830b9bff12a3359791c"
server_key_version: "encryptionkey@0"
I20260925 02:49:32.586834 27676 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:49:32.590217 27691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:32.591588 27676 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:49:32.591773 27676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "ac1c2bfa50a443cabcb393b10c547761"
format_stamp: "Formatted at 2026-09-25 02:49:32 on dist-test-slave-n563"
server_key: "203f9aad80b354002acb162e1cf0d7fc"
server_key_iv: "348ba0e331d81830b9bff12a3359791c"
server_key_version: "encryptionkey@0"
I20260925 02:49:32.591974 27676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:32.607841 27676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:32.611904 27676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:32.612210 27676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:32.631361 27676 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:41875
I20260925 02:49:32.631386 27742 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:41875 every 8 connection(s)
I20260925 02:49:32.632822 27676 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:32.634563 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 27676
I20260925 02:49:32.634768 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:32.635243 5346 external_mini_cluster.cc:1428] Setting key 0a15b087aa997e2a00e13c0436dafdd6
I20260925 02:49:32.637209 27743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 25 02:49:32 dist-test-slave-n563 krb5kdc[27663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:49:32.651095 27743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761: Bootstrap starting.
I20260925 02:49:32.654644 27743 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:32.655728 27743 log.cc:824] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:32.658795 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:32.637034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39247 (local address 127.5.56.190:41875)
0925 02:49:32.637864 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:32.637876 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:32.637917 (+ 41us) server_negotiation.cc:407] Connection header received
0925 02:49:32.639643 (+ 1726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:32.639664 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:32.640074 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:32.640417 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:32.642235 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:32.643662 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:32.645465 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:32.646108 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:32.646251 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:32.651038 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:32.651046 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:32.651061 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:32.651102 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:32.654177 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:32.655011 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:32.655017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:32.655024 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:32.655196 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:32.655593 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:32.655598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:32.655607 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:32.656231 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:32.656573 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:32.656957 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 02:49:32.657436 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":400,"threads_started":1}
I20260925 02:49:32.659271 27743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761: No bootstrap required, opened a new log
I20260925 02:49:32.662597 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:32.668108 27743 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c2bfa50a443cabcb393b10c547761" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41875 } }
I20260925 02:49:32.668591 27743 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:32.668699 27743 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac1c2bfa50a443cabcb393b10c547761, State: Initialized, Role: FOLLOWER
I20260925 02:49:32.669420 27743 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c2bfa50a443cabcb393b10c547761" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41875 } }
I20260925 02:49:32.669716 27743 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:32.669899 27743 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:32.670099 27743 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:32.673194 27743 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c2bfa50a443cabcb393b10c547761" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41875 } }
I20260925 02:49:32.673810 27743 leader_election.cc:302] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac1c2bfa50a443cabcb393b10c547761; no voters:
I20260925 02:49:32.674512 27743 leader_election.cc:288] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:32.674664 27753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:32.676231 27753 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [term 1 LEADER]: Becoming Leader. State: Replica: ac1c2bfa50a443cabcb393b10c547761, State: Running, Role: LEADER
I20260925 02:49:32.676934 27743 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:32.676980 27753 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c2bfa50a443cabcb393b10c547761" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41875 } }
I20260925 02:49:32.680287 27755 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac1c2bfa50a443cabcb393b10c547761. Latest consensus state: current_term: 1 leader_uuid: "ac1c2bfa50a443cabcb393b10c547761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c2bfa50a443cabcb393b10c547761" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41875 } } }
I20260925 02:49:32.680662 27755 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:32.681133 27754 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac1c2bfa50a443cabcb393b10c547761" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac1c2bfa50a443cabcb393b10c547761" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 41875 } } }
I20260925 02:49:32.681427 27754 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:32.686147 27746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:32.666793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50687 (local address 127.5.56.190:41875)
0925 02:49:32.666999 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:32.667008 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:32.667536 (+ 528us) server_negotiation.cc:407] Connection header received
0925 02:49:32.667701 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:32.667704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:32.667766 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:32.667843 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:32.669797 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:32.670785 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:32.672545 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:32.672957 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:32.673047 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:32.676089 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:32.676102 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:32.676110 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:32.676158 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:32.680671 (+ 4513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:32.681858 (+ 1187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:32.681865 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:32.681870 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:32.682091 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:32.682744 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:32.682752 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:32.682770 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:32.683336 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:32.685122 (+ 1786us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:32.685333 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 02:49:32.685630 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260925 02:49:32.687924 27759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:32.693195 27759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:32.703521 27759 catalog_manager.cc:1347] Generated new cluster ID: 1fb153663c544c4d90ee549b77bdb152
I20260925 02:49:32.703631 27759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:32.739591 27759 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:32.741130 27759 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:32.749327 27759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac1c2bfa50a443cabcb393b10c547761: Generated new TSK 0
I20260925 02:49:32.750201 27759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:32.756785 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27676
I20260925 02:49:32.757939 27738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:32.850236 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 27676
2026-09-25T02:49:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2080 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:32 dist-test-slave-n563 krb5kdc[27775](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:32 dist-test-slave-n563 krb5kdc[27775](info): set up 2 sockets
Sep 25 02:49:32 dist-test-slave-n563 krb5kdc[27775](info): commencing operation
krb5kdc: starting...
W20260925 02:49:33.968004 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.083s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:34 dist-test-slave-n563 krb5kdc[27775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304574, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:49:34.241734 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:34541
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:35371
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:34541
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:34.579691 27788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:34.580073 27788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:34.580231 27788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:34.589176 27788 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:34.589329 27788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:34.589432 27788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:34.589524 27788 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:34.589589 27788 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:34.599933 27788 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:35371
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:34541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:34541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:34.601310 27788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:34.602713 27788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:34.614614 27794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:34.614727 27793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:34.614724 27796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:34.615792 27788 server_base.cc:1034] running on GCE node
I20260925 02:49:34.616570 27788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:34.618005 27788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:34.619207 27788 hybrid_clock.cc:648] HybridClock initialized: now 1790304574619166 us; error 27 us; skew 500 ppm
I20260925 02:49:34.619525 27788 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:34 dist-test-slave-n563 krb5kdc[27775](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304574, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:34.628319 27788 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:49:34.629911 27788 webserver.cc:460] Webserver started at http://127.5.56.190:45863/ using document root <none> and password file <none>
I20260925 02:49:34.630683 27788 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:34.630802 27788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:34.631062 27788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:34.634080 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "b4ab8d1c57ce49a88c935db42f99b700"
format_stamp: "Formatted at 2026-09-25 02:49:34 on dist-test-slave-n563"
server_key: "ffe2bea7f71147a1674b61fc5c245240"
server_key_iv: "d5c2e11f4d870596294ca5a131940bf7"
server_key_version: "encryptionkey@0"
I20260925 02:49:34.634877 27788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "b4ab8d1c57ce49a88c935db42f99b700"
format_stamp: "Formatted at 2026-09-25 02:49:34 on dist-test-slave-n563"
server_key: "ffe2bea7f71147a1674b61fc5c245240"
server_key_iv: "d5c2e11f4d870596294ca5a131940bf7"
server_key_version: "encryptionkey@0"
I20260925 02:49:34.640082 27788 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260925 02:49:34.643581 27803 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:34.644959 27788 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260925 02:49:34.645149 27788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "b4ab8d1c57ce49a88c935db42f99b700"
format_stamp: "Formatted at 2026-09-25 02:49:34 on dist-test-slave-n563"
server_key: "ffe2bea7f71147a1674b61fc5c245240"
server_key_iv: "d5c2e11f4d870596294ca5a131940bf7"
server_key_version: "encryptionkey@0"
I20260925 02:49:34.645352 27788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:34.695534 27788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:34.699620 27788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:34.699918 27788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:34.718581 27788 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:34541
I20260925 02:49:34.718657 27854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:34541 every 8 connection(s)
I20260925 02:49:34.720325 27788 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:34.724153 27855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:34.729204 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 27788
I20260925 02:49:34.729435 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:34.729882 5346 external_mini_cluster.cc:1428] Setting key d5c8948ddd3b6d8b4d614bd6760e786a
I20260925 02:49:34.735271 27855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700: Bootstrap starting.
I20260925 02:49:34.738492 27855 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:34.739877 27855 log.cc:824] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:34.743408 27855 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700: No bootstrap required, opened a new log
I20260925 02:49:34.743945 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:34.731844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53987 (local address 127.5.56.190:34541)
0925 02:49:34.732718 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:34.732732 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:34.732787 (+ 55us) server_negotiation.cc:407] Connection header received
0925 02:49:34.734687 (+ 1900us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:34.734713 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:34.735118 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:34.735502 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:34.736058 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:34.737369 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:34.738758 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:34.739853 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:34.740051 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:34.741304 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:34.741355 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:34.741374 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:34.742091 (+ 717us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:34.742186 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:34.742217 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:34.742361 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:34.742606 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 02:49:34.742888 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":390,"threads_started":1}
I20260925 02:49:34.748147 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:34.752452 27855 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab8d1c57ce49a88c935db42f99b700" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34541 } }
I20260925 02:49:34.753099 27855 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:34.753189 27855 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4ab8d1c57ce49a88c935db42f99b700, State: Initialized, Role: FOLLOWER
I20260925 02:49:34.753935 27855 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab8d1c57ce49a88c935db42f99b700" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34541 } }
I20260925 02:49:34.754210 27855 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:34.754366 27855 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:34.754611 27855 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:34.756804 27855 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab8d1c57ce49a88c935db42f99b700" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34541 } }
I20260925 02:49:34.757350 27855 leader_election.cc:302] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4ab8d1c57ce49a88c935db42f99b700; no voters:
I20260925 02:49:34.758008 27855 leader_election.cc:288] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:34.758123 27865 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:34.759964 27858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:34.752760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43073 (local address 127.5.56.190:34541)
0925 02:49:34.752952 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:34.752959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:34.753367 (+ 408us) server_negotiation.cc:407] Connection header received
0925 02:49:34.753588 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:34.753594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:34.753686 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:34.753787 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:34.754404 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:34.755212 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:34.756474 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:34.756929 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:34.757081 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:34.758328 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:34.758334 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:34.758341 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:34.758364 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:34.758448 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:34.758478 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:34.759275 (+ 797us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:34.759336 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 02:49:34.759459 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:49:34.760221 27855 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:34.762735 27865 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [term 1 LEADER]: Becoming Leader. State: Replica: b4ab8d1c57ce49a88c935db42f99b700, State: Running, Role: LEADER
I20260925 02:49:34.763263 27865 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab8d1c57ce49a88c935db42f99b700" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34541 } }
I20260925 02:49:34.766065 27866 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4ab8d1c57ce49a88c935db42f99b700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab8d1c57ce49a88c935db42f99b700" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34541 } } }
I20260925 02:49:34.766564 27866 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:34.766078 27867 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4ab8d1c57ce49a88c935db42f99b700. Latest consensus state: current_term: 1 leader_uuid: "b4ab8d1c57ce49a88c935db42f99b700" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4ab8d1c57ce49a88c935db42f99b700" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34541 } } }
I20260925 02:49:34.767051 27867 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700 [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:34.769917 27873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:34.776433 27873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:34.785956 27873 catalog_manager.cc:1347] Generated new cluster ID: 4d7b1dd6f522421fbccfd1c320b0a51e
I20260925 02:49:34.786100 27873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:34.802280 27873 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:34.804399 27873 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:34.817466 27873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4ab8d1c57ce49a88c935db42f99b700: Generated new TSK 0
I20260925 02:49:34.818356 27873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:34.825119 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27788
W20260925 02:49:34.826241 27850 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53987
W20260925 02:49:34.826972 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27788: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:34.827111 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 27788
2026-09-25T02:49:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1976 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:34 dist-test-slave-n563 krb5kdc[27886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:34 dist-test-slave-n563 krb5kdc[27886](info): set up 2 sockets
Sep 25 02:49:34 dist-test-slave-n563 krb5kdc[27886](info): commencing operation
krb5kdc: starting...
W20260925 02:49:35.956089 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:36 dist-test-slave-n563 krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:49:36.230366 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:35111
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:35111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:36.570745 27900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:36.571043 27900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:36.571218 27900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:36.578534 27900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:36.578646 27900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:36.578750 27900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:36.578853 27900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:36.578940 27900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:36.588888 27900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:35111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:35111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:36.590349 27900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:36.592329 27900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:36.604705 27905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:36.604728 27906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:36.604660 27908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:36.605520 27900 server_base.cc:1034] running on GCE node
I20260925 02:49:36.606314 27900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:36.607805 27900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:36.609020 27900 hybrid_clock.cc:648] HybridClock initialized: now 1790304576608974 us; error 31 us; skew 500 ppm
I20260925 02:49:36.609406 27900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:36 dist-test-slave-n563 krb5kdc[27886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304576, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:36.612639 27900 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:49:36.614214 27900 webserver.cc:460] Webserver started at http://127.5.56.190:37913/ using document root <none> and password file <none>
I20260925 02:49:36.615000 27900 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:36.615140 27900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:36.615481 27900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:36.618885 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "940741d5cea046cc90a53c53091c859e"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-n563"
server_key: "00d869d11b2b9b1dfb2f651fa3963274"
server_key_iv: "95e01606d8ca9e6abd728519f1f2eee1"
server_key_version: "encryptionkey@0"
I20260925 02:49:36.619730 27900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "940741d5cea046cc90a53c53091c859e"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-n563"
server_key: "00d869d11b2b9b1dfb2f651fa3963274"
server_key_iv: "95e01606d8ca9e6abd728519f1f2eee1"
server_key_version: "encryptionkey@0"
I20260925 02:49:36.625250 27900 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:49:36.628890 27915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:36.630203 27900 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260925 02:49:36.630380 27900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "940741d5cea046cc90a53c53091c859e"
format_stamp: "Formatted at 2026-09-25 02:49:36 on dist-test-slave-n563"
server_key: "00d869d11b2b9b1dfb2f651fa3963274"
server_key_iv: "95e01606d8ca9e6abd728519f1f2eee1"
server_key_version: "encryptionkey@0"
I20260925 02:49:36.630631 27900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:36.637112 27900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:36.641207 27900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:36.641508 27900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:36.660578 27900 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:35111
I20260925 02:49:36.660621 27967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:35111 every 8 connection(s)
I20260925 02:49:36.662083 27900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:36.665928 27968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:36.667348 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 27900
I20260925 02:49:36.667523 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:36.668099 5346 external_mini_cluster.cc:1428] Setting key 2af243fb3101b137d1054f3589bc185e
I20260925 02:49:36.676404 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:36.670205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35193 (local address 127.5.56.190:35111)
0925 02:49:36.671381 (+ 1176us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:36.671392 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:36.671439 (+ 47us) server_negotiation.cc:407] Connection header received
0925 02:49:36.673039 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:36.673066 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:36.673420 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:36.673736 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:36.674819 (+ 1083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:36.674825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:36.674838 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:36.674851 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:36.674908 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:36.674927 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:36.675101 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:36.675205 (+ 104us) server_negotiation.cc:299] Negotiation successful
0925 02:49:36.675451 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":932,"thread_start_us":609,"threads_started":1}
I20260925 02:49:36.679553 27968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e: Bootstrap starting.
I20260925 02:49:36.680197 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:36.683697 27968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:36.685431 27968 log.cc:824] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:36.689227 27968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e: No bootstrap required, opened a new log
I20260925 02:49:36.689888 27971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:36.686590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45011 (local address 127.5.56.190:35111)
0925 02:49:36.686791 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:36.686796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:36.687877 (+ 1081us) server_negotiation.cc:407] Connection header received
0925 02:49:36.688099 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:36.688104 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:36.688151 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:36.688261 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:36.689196 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:36.689202 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:36.689207 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 02:49:36.689229 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:36.689279 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 02:49:36.689318 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:36.689441 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:36.689534 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 02:49:36.689595 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260925 02:49:36.698740 27968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940741d5cea046cc90a53c53091c859e" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35111 } }
I20260925 02:49:36.699215 27968 raft_consensus.cc:374] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 02:49:36.699360 27968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 940741d5cea046cc90a53c53091c859e, State: Initialized, Role: FOLLOWER
I20260925 02:49:36.700127 27968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940741d5cea046cc90a53c53091c859e" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35111 } }
I20260925 02:49:36.700405 27968 raft_consensus.cc:388] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 02:49:36.700590 27968 raft_consensus.cc:482] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 02:49:36.700814 27968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:36.703781 27968 raft_consensus.cc:504] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940741d5cea046cc90a53c53091c859e" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35111 } }
I20260925 02:49:36.704380 27968 leader_election.cc:302] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 940741d5cea046cc90a53c53091c859e; no voters:
I20260925 02:49:36.705024 27968 leader_election.cc:288] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 02:49:36.705199 27978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:36.705906 27978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [term 1 LEADER]: Becoming Leader. State: Replica: 940741d5cea046cc90a53c53091c859e, State: Running, Role: LEADER
I20260925 02:49:36.706532 27978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940741d5cea046cc90a53c53091c859e" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35111 } }
I20260925 02:49:36.707675 27968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:36.709842 27980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 940741d5cea046cc90a53c53091c859e. Latest consensus state: current_term: 1 leader_uuid: "940741d5cea046cc90a53c53091c859e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940741d5cea046cc90a53c53091c859e" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35111 } } }
I20260925 02:49:36.710264 27980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:36.709941 27979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "940741d5cea046cc90a53c53091c859e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "940741d5cea046cc90a53c53091c859e" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35111 } } }
I20260925 02:49:36.710489 27979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:36.713392 27984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:36.718931 27984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:36.737908 27984 catalog_manager.cc:1347] Generated new cluster ID: ca04db204a414741825b80a582ca3d63
I20260925 02:49:36.738044 27984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:36.774943 27984 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:36.775393 27984 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:36.784370 27984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 940741d5cea046cc90a53c53091c859e: Generated new TSK 0
I20260925 02:49:36.785274 27984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:36.790403 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27900
W20260925 02:49:36.791574 27963 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:35193
W20260925 02:49:36.792384 5346 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 27900: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260925 02:49:36.792524 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 27900
2026-09-25T02:49:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1967 ms)
[----------] 18 tests from AuthTokenIssuingTest (36277 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:36 dist-test-slave-n563 krb5kdc[28000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:36 dist-test-slave-n563 krb5kdc[28000](info): set up 2 sockets
Sep 25 02:49:36 dist-test-slave-n563 krb5kdc[28000](info): commencing operation
krb5kdc: starting...
W20260925 02:49:37.904173 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:38 dist-test-slave-n563 krb5kdc[28000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:49:38.186604 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:34775
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:34775,127.5.56.189:39407,127.5.56.188:35383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:38.524469 28013 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:49:38.524833 28013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:38.524983 28013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:38.525134 28013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:38.532464 28013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:38.532552 28013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:38.532666 28013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:38.532742 28013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:38.532837 28013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:38.542774 28013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:34775,127.5.56.189:39407,127.5.56.188:35383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:34775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:38.544186 28013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:38.545534 28013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:38.558164 28021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:38.558250 28019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:38.558755 28018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:38.559080 28013 server_base.cc:1034] running on GCE node
I20260925 02:49:38.559895 28013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:38.561349 28013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:38.562590 28013 hybrid_clock.cc:648] HybridClock initialized: now 1790304578562536 us; error 31 us; skew 500 ppm
I20260925 02:49:38.562927 28013 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:38 dist-test-slave-n563 krb5kdc[28000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304578, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:38.566171 28013 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:49:38.567803 28013 webserver.cc:460] Webserver started at http://127.5.56.190:42983/ using document root <none> and password file <none>
I20260925 02:49:38.568507 28013 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:38.568639 28013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:38.568969 28013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:38.572050 28013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b"
format_stamp: "Formatted at 2026-09-25 02:49:38 on dist-test-slave-n563"
server_key: "17ee96e2e9d2f4689cc93581e07d56ef"
server_key_iv: "43f316c79774214ac8517cdaa46be9d6"
server_key_version: "encryptionkey@0"
I20260925 02:49:38.572862 28013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b"
format_stamp: "Formatted at 2026-09-25 02:49:38 on dist-test-slave-n563"
server_key: "17ee96e2e9d2f4689cc93581e07d56ef"
server_key_iv: "43f316c79774214ac8517cdaa46be9d6"
server_key_version: "encryptionkey@0"
I20260925 02:49:38.578092 28013 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260925 02:49:38.581691 28028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:38.583190 28013 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:49:38.583392 28013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b"
format_stamp: "Formatted at 2026-09-25 02:49:38 on dist-test-slave-n563"
server_key: "17ee96e2e9d2f4689cc93581e07d56ef"
server_key_iv: "43f316c79774214ac8517cdaa46be9d6"
server_key_version: "encryptionkey@0"
I20260925 02:49:38.583618 28013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:38.604964 28013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:38.609061 28013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:38.609345 28013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:38.627147 28013 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:34775
I20260925 02:49:38.627168 28080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:34775 every 8 connection(s)
I20260925 02:49:38.628556 28013 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:38.632308 28081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:38.633134 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 28013
I20260925 02:49:38.633368 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:38.634179 5346 external_mini_cluster.cc:1428] Setting key 3dc4bcc8c3f8de42b6e31fabca577cc5
I20260925 02:49:38.642525 28081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:49:38 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304578, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.5.56.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.189@KRBTEST.COM" created.
I20260925 02:49:38.668958 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:38.646732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54720 (local address 127.5.56.190:34775)
0925 02:49:38.647797 (+ 1065us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:38.647807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:38.648015 (+ 208us) server_negotiation.cc:407] Connection header received
0925 02:49:38.651227 (+ 3212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:38.651233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:38.651485 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:38.651656 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:38.653361 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:38.654651 (+ 1290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:38.656842 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:38.657428 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:38.657551 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:38.661875 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:38.661883 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:38.661896 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:38.661929 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:38.664764 (+ 2835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:38.665673 (+ 909us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:38.665679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:38.665681 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:38.665825 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:38.666285 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:38.666288 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:38.666290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:38.666866 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:38.667023 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:38.667601 (+ 578us) server_negotiation.cc:299] Negotiation successful
0925 02:49:38.667834 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":900,"thread_start_us":535,"threads_started":1}
I20260925 02:49:38.669032 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:38.646733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:34775 (local address 127.0.0.1:54720)
0925 02:49:38.647853 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:38.647902 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:38.650795 (+ 2893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:38.651740 (+ 945us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:38.651749 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:38.652624 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:38.653135 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:38.653162 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:38.654807 (+ 1645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:38.654815 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:38.656699 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:38.656717 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:38.657674 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:38.657683 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:38.657903 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:38.658681 (+ 778us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:38.658715 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:38.661668 (+ 2953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:38.664926 (+ 3258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:38.664941 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:38.664955 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:38.665553 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:38.665943 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:38.665948 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:38.665952 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:38.666174 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:38.667001 (+ 827us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:38.667007 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:38.667210 (+ 203us) client_negotiation.cc:769] Sending connection context
0925 02:49:38.667480 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 02:49:38.667992 (+ 512us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":124,"thread_start_us":525,"threads_started":1}
I20260925 02:49:38.671939 28081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:49:38.673094 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:38.672593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48320)
0925 02:49:38.672804 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:38.672930 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260925 02:49:38.673355 28030 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
I20260925 02:49:38.675036 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:38.674565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48334)
0925 02:49:38.674764 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:38.674861 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:49:38.675235 28081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:39407: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab.
Entry for principal kudu/127.5.56.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab.
I20260925 02:49:38.701159 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.189:39407
--webserver_interface=127.5.56.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab
--principal=kudu/127.5.56.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:34775,127.5.56.189:39407,127.5.56.188:35383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:49:38.724639 28081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } attempt: 1
I20260925 02:49:38.726060 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:38.725431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48346)
0925 02:49:38.725693 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:38.725840 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260925 02:49:38.727202 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:38.726769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48360)
0925 02:49:38.726931 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:38.727015 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 02:49:38.727416 28081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:39407: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
I20260925 02:49:38.795691 28081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } attempt: 2
I20260925 02:49:38.796947 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:38.796407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48366)
0925 02:49:38.796651 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:38.796763 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 02:49:38.798113 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:38.797631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48378)
0925 02:49:38.797837 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:38.797946 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 02:49:38.798308 28081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:39407: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
I20260925 02:49:38.889644 28081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } attempt: 3
I20260925 02:49:38.890942 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:38.890336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48382)
0925 02:49:38.890629 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:38.890744 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260925 02:49:38.892041 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:38.891608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48384)
0925 02:49:38.891766 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:38.891851 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:49:38.892244 28081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:39407: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
I20260925 02:49:39.035480 28081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } attempt: 4
I20260925 02:49:39.036752 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.036154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48394)
0925 02:49:39.036479 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.036584 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260925 02:49:39.037773 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.037383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48404)
0925 02:49:39.037533 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.037625 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:49:39.037966 28081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:39407: Network error: Client connection negotiation failed: client connection to 127.5.56.189:39407: connect: Connection refused (error 111)
W20260925 02:49:39.039415 28088 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:49:39.039775 28088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:39.039880 28088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:39.040012 28088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:39.047585 28088 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:39.047683 28088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:39.047760 28088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:39.047834 28088 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:39.047945 28088 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:39.058799 28088 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:34775,127.5.56.189:39407,127.5.56.188:35383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab
--principal=kudu/127.5.56.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.189:39407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.56.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:39.060258 28088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:39.061710 28088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:39.074235 28094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:39.074239 28093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:39.074235 28096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:39.075243 28088 server_base.cc:1034] running on GCE node
I20260925 02:49:39.076054 28088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:39.077463 28088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:39.078711 28088 hybrid_clock.cc:648] HybridClock initialized: now 1790304579078621 us; error 46 us; skew 500 ppm
I20260925 02:49:39.079154 28088 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:39.082465 28088 init.cc:377] Logged in from keytab as kudu/127.5.56.189@KRBTEST.COM (short username kudu)
I20260925 02:49:39.083940 28088 webserver.cc:460] Webserver started at http://127.5.56.189:43235/ using document root <none> and password file <none>
I20260925 02:49:39.084652 28088 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:39.084789 28088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:39.085126 28088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:39.088392 28088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/data/instance:
uuid: "5917a13320a5485cb05263bd4cb65e21"
format_stamp: "Formatted at 2026-09-25 02:49:39 on dist-test-slave-n563"
server_key: "96da61fe4bb14d7f1e777fcf7f57030e"
server_key_iv: "a9f8c7886f4bc0cfc98ed1a8cb48a410"
server_key_version: "encryptionkey@0"
I20260925 02:49:39.089174 28088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/wal/instance:
uuid: "5917a13320a5485cb05263bd4cb65e21"
format_stamp: "Formatted at 2026-09-25 02:49:39 on dist-test-slave-n563"
server_key: "96da61fe4bb14d7f1e777fcf7f57030e"
server_key_iv: "a9f8c7886f4bc0cfc98ed1a8cb48a410"
server_key_version: "encryptionkey@0"
I20260925 02:49:39.094368 28088 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260925 02:49:39.097757 28103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:39.099144 28088 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260925 02:49:39.099323 28088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/wal
uuid: "5917a13320a5485cb05263bd4cb65e21"
format_stamp: "Formatted at 2026-09-25 02:49:39 on dist-test-slave-n563"
server_key: "96da61fe4bb14d7f1e777fcf7f57030e"
server_key_iv: "a9f8c7886f4bc0cfc98ed1a8cb48a410"
server_key_version: "encryptionkey@0"
I20260925 02:49:39.099548 28088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:39.122592 28088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:39.126839 28088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:39.127131 28088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:39.146103 28088 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.189:39407
I20260925 02:49:39.146124 28154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.189:39407 every 8 connection(s)
I20260925 02:49:39.147501 28088 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
I20260925 02:49:39.151242 28155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:39.157709 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 28088
I20260925 02:49:39.157918 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-1/wal/instance
I20260925 02:49:39.158485 5346 external_mini_cluster.cc:1428] Setting key bcf04bd4619b6755345d55e5557d2924
I20260925 02:49:39.157858 28155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:49:39.182790 28084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.161385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54732 (local address 127.5.56.190:34775)
0925 02:49:39.161627 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.161635 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.162894 (+ 1259us) server_negotiation.cc:407] Connection header received
0925 02:49:39.165804 (+ 2910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.165809 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.165894 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.165999 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.168174 (+ 2175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.168970 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.170958 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.171373 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.171474 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.176158 (+ 4684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.176163 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.176166 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.176201 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.179121 (+ 2920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.180194 (+ 1073us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.180199 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.180201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.180339 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.180886 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.180890 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.180892 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.181186 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.181314 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.182301 (+ 987us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.182556 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 02:49:39.184506 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.161812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:34775 (local address 127.0.0.1:54732)
0925 02:49:39.162705 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.162756 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:39.165456 (+ 2700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:39.166273 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:39.166287 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:39.167114 (+ 827us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:39.167946 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.167975 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.169151 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.169163 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.170794 (+ 1631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.170814 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.171622 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.171631 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.171975 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.172729 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:39.172770 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:39.175892 (+ 3122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:39.179302 (+ 3410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.179315 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.179339 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.180054 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.180455 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.180463 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.180470 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.180760 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.181326 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:39.181337 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:39.181863 (+ 526us) client_negotiation.cc:769] Sending connection context
0925 02:49:39.182201 (+ 338us) client_negotiation.cc:240] Negotiation successful
0925 02:49:39.182844 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":348,"threads_started":1}
I20260925 02:49:39.186465 28155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.56.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.188@KRBTEST.COM" created.
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
I20260925 02:49:39.203115 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.187390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48420)
0925 02:49:39.187573 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.187588 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:39.187670 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:39.188893 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:39.188897 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:39.189342 (+ 445us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:39.189527 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.189536 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.191169 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.191176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.192573 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.192592 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.193488 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.193497 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.193714 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.194327 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:39.194354 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:39.196927 (+ 2573us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:39.200198 (+ 3271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.200206 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.200211 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.200737 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.201137 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.201143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.201148 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.201336 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.202131 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:39.202136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:39.202263 (+ 127us) client_negotiation.cc:769] Sending connection context
0925 02:49:39.202466 (+ 203us) client_negotiation.cc:240] Negotiation successful
0925 02:49:39.202823 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 02:49:39.203632 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.187707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48420 (local address 127.5.56.189:39407)
0925 02:49:39.188460 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.188469 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.188499 (+ 30us) server_negotiation.cc:407] Connection header received
0925 02:49:39.188553 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.188559 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.188738 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.188902 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.189711 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.191032 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.192744 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.193319 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.193425 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.197145 (+ 3720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.197158 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.197174 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.197209 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.200055 (+ 2846us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.200890 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.200897 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.200899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.201018 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.201444 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.201448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.201450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.201939 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.202157 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.202714 (+ 557us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.203043 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":415,"threads_started":1}
I20260925 02:49:39.206297 28155 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:49:39.207597 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.207028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38560)
0925 02:49:39.207249 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.207406 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260925 02:49:39.207876 28107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
I20260925 02:49:39.209612 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.209150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38566)
0925 02:49:39.209335 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.209463 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 02:49:39.209852 28155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:35383: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab.
Entry for principal kudu/127.5.56.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab.
I20260925 02:49:39.226373 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.188:35383
--webserver_interface=127.5.56.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab
--principal=kudu/127.5.56.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:33287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:34775,127.5.56.189:39407,127.5.56.188:35383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:49:39.259179 28155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } attempt: 1
I20260925 02:49:39.260321 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.259858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38578)
0925 02:49:39.260053 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.260165 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260925 02:49:39.261336 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.261010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38580)
0925 02:49:39.261161 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.261238 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:49:39.261552 28155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:35383: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
I20260925 02:49:39.329772 28155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } attempt: 2
I20260925 02:49:39.331049 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.330584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38588)
0925 02:49:39.330805 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.330897 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260925 02:49:39.332094 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.331761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38594)
0925 02:49:39.331918 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.331986 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:49:39.332297 28155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:35383: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
I20260925 02:49:39.337239 28081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } attempt: 5
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304578, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
I20260925 02:49:39.351871 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.337969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48430)
0925 02:49:39.338156 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.338181 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:39.338292 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:39.338759 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:39.338765 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:39.339149 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:39.339331 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.339340 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.340834 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.340838 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.341855 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.341870 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.343053 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.343057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.343216 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.343859 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:39.343877 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:39.346032 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:39.349645 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.349649 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.349652 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.350011 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.350456 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.350459 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.350461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.350583 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.351178 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:39.351181 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:39.351328 (+ 147us) client_negotiation.cc:769] Sending connection context
0925 02:49:39.351434 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 02:49:39.351630 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260925 02:49:39.352180 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.338016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48430 (local address 127.5.56.189:39407)
0925 02:49:39.338184 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.338190 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.338260 (+ 70us) server_negotiation.cc:407] Connection header received
0925 02:49:39.338432 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.338438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.338534 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.338654 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.339507 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.340697 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.342018 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.342895 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.343054 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.346228 (+ 3174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.346235 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.346240 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.346278 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.349518 (+ 3240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.350133 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.350141 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.350146 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.350290 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.350688 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.350693 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.350697 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.351062 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.351202 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.351578 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.351823 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260925 02:49:39.353129 28081 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:49:39.354136 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.353757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38602)
0925 02:49:39.353920 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.354008 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260925 02:49:39.355015 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.354713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38610)
0925 02:49:39.354849 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.354918 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260925 02:49:39.355229 28081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:35383: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
I20260925 02:49:39.406443 28081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } attempt: 1
I20260925 02:49:39.407647 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.407179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38622)
0925 02:49:39.407388 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.407479 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260925 02:49:39.408602 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.408261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38628)
0925 02:49:39.408430 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.408499 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:49:39.408818 28081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:35383: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
I20260925 02:49:39.423512 28155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } attempt: 3
I20260925 02:49:39.424777 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.424276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38636)
0925 02:49:39.424530 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.424620 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260925 02:49:39.426100 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.425669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38650)
0925 02:49:39.425920 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.425991 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149}
W20260925 02:49:39.426313 28155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:35383: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
I20260925 02:49:39.477020 28081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } attempt: 2
I20260925 02:49:39.478221 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.477716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38652)
0925 02:49:39.477949 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.478061 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260925 02:49:39.479224 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.478894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38656)
0925 02:49:39.479041 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.479114 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:49:39.479414 28081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:35383: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
I20260925 02:49:39.569568 28155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } attempt: 4
I20260925 02:49:39.570740 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.570223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38668)
0925 02:49:39.570493 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.570581 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 02:49:39.571787 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.571464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38676)
0925 02:49:39.571615 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.571686 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260925 02:49:39.571993 28155 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:35383: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
W20260925 02:49:39.576993 28162 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:49:39.577418 28162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:39.577561 28162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:39.577701 28162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:39.585367 28162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:39.585532 28162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:39.585656 28162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:39.585760 28162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
I20260925 02:49:39.585649 28081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } attempt: 3
W20260925 02:49:39.585870 28162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:39.587095 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.586329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38682)
0925 02:49:39.586783 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.586917 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":274}
I20260925 02:49:39.588219 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.587816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38684)
0925 02:49:39.588035 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.588116 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 02:49:39.588451 28081 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:35383: Network error: Client connection negotiation failed: client connection to 127.5.56.188:35383: connect: Connection refused (error 111)
I20260925 02:49:39.596935 28162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:33287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:34775,127.5.56.189:39407,127.5.56.188:35383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab
--principal=kudu/127.5.56.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.188:35383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.56.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:39.598352 28162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:39.599824 28162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:39.613108 28168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:39.613143 28170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:39.613189 28167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:39.619426 28162 server_base.cc:1034] running on GCE node
I20260925 02:49:39.620316 28162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:39.621915 28162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:39.623137 28162 hybrid_clock.cc:648] HybridClock initialized: now 1790304579623073 us; error 38 us; skew 500 ppm
I20260925 02:49:39.623510 28162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:39.626803 28162 init.cc:377] Logged in from keytab as kudu/127.5.56.188@KRBTEST.COM (short username kudu)
I20260925 02:49:39.628553 28162 webserver.cc:460] Webserver started at http://127.5.56.188:33013/ using document root <none> and password file <none>
I20260925 02:49:39.629297 28162 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:39.629441 28162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:39.629805 28162 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:39.632840 28162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/data/instance:
uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b"
format_stamp: "Formatted at 2026-09-25 02:49:39 on dist-test-slave-n563"
server_key: "b01ed306d92ec8725c5cc8444291460a"
server_key_iv: "7dc4c4bf011d4aa08958441ede2d456e"
server_key_version: "encryptionkey@0"
I20260925 02:49:39.633634 28162 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/wal/instance:
uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b"
format_stamp: "Formatted at 2026-09-25 02:49:39 on dist-test-slave-n563"
server_key: "b01ed306d92ec8725c5cc8444291460a"
server_key_iv: "7dc4c4bf011d4aa08958441ede2d456e"
server_key_version: "encryptionkey@0"
I20260925 02:49:39.638805 28162 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260925 02:49:39.642520 28177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:39.644404 28162 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260925 02:49:39.644598 28162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/wal
uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b"
format_stamp: "Formatted at 2026-09-25 02:49:39 on dist-test-slave-n563"
server_key: "b01ed306d92ec8725c5cc8444291460a"
server_key_iv: "7dc4c4bf011d4aa08958441ede2d456e"
server_key_version: "encryptionkey@0"
I20260925 02:49:39.644833 28162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:39.666523 28162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:39.670629 28162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:39.670921 28162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:39.690021 28162 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.188:35383
I20260925 02:49:39.690042 28229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.188:35383 every 8 connection(s)
I20260925 02:49:39.691512 28162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
I20260925 02:49:39.695422 28230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:39.695393 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 28162
I20260925 02:49:39.695585 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790304212743187-5346-0/minicluster-data/master-2/wal/instance
I20260925 02:49:39.696210 5346 external_mini_cluster.cc:1428] Setting key 9a34f92cf304e2587676e26e68bb6c20
I20260925 02:49:39.705466 28230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:49:39.720535 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.698121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38686 (local address 127.5.56.188:35383)
0925 02:49:39.698962 (+ 841us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.698973 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.699015 (+ 42us) server_negotiation.cc:407] Connection header received
0925 02:49:39.700607 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.700629 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.701023 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.701378 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.703198 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.704580 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.706357 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.707102 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.707231 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.712200 (+ 4969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.712210 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.712224 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.712265 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.715577 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.716450 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.716457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.716463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.716648 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.717097 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.717102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.717106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.717823 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.718052 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.718618 (+ 566us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.719062 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":414,"threads_started":1}
I20260925 02:49:39.724874 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.708516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.5.56.190:34775)
0925 02:49:39.709894 (+ 1378us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.709898 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.709913 (+ 15us) server_negotiation.cc:407] Connection header received
0925 02:49:39.709962 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.709965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.710021 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.710140 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.711558 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.712575 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.714091 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.714585 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.714726 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.718461 (+ 3735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.718468 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.718472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.718513 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.721835 (+ 3322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.722865 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.722869 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.722871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.722992 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.723389 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.723397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.723402 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.723672 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.723834 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.724374 (+ 540us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.724571 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1265,"thread_start_us":1149,"threads_started":1}
I20260925 02:49:39.725255 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.708528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:34775 (local address 127.0.0.1:54746)
0925 02:49:39.709255 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.709301 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:39.709470 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:39.710207 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:39.710214 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:39.711101 (+ 887us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:39.711402 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.711411 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.712760 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.712807 (+ 47us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.713930 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.713941 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.714777 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.714783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.714957 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.715538 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:39.715554 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:39.718227 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:39.721974 (+ 3747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.721985 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.721995 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.722705 (+ 710us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.723104 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.723108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.723111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.723272 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.723810 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:39.723818 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:39.724032 (+ 214us) client_negotiation.cc:769] Sending connection context
0925 02:49:39.724233 (+ 201us) client_negotiation.cc:240] Negotiation successful
0925 02:49:39.724660 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":371,"threads_started":1}
I20260925 02:49:39.726984 28230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
I20260925 02:49:39.740655 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.724049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54756 (local address 127.5.56.190:34775)
0925 02:49:39.725105 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.725110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.725126 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:49:39.725168 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.725174 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.725264 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.725372 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.727314 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.728205 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.729795 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.730309 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.730474 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.735133 (+ 4659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.735139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.735144 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.735184 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.737791 (+ 2607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.738586 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.738595 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.738599 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.738733 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.739092 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.739100 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.739104 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.739466 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.739761 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.740042 (+ 281us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.740280 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":962,"thread_start_us":1540,"threads_started":1}
I20260925 02:49:39.741046 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.727615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:39407 (local address 127.0.0.1:48432)
0925 02:49:39.727839 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.727857 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:39.727950 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:39.728322 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:39.728328 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:39.728662 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:39.728799 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.728806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.730075 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.730082 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.731144 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.731156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.732122 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.732126 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.732257 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.732809 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:39.732825 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:39.735144 (+ 2319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:39.738507 (+ 3363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.738511 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.738514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.739029 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.739413 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.739419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.739424 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.739601 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.740196 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:39.740203 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:39.740374 (+ 171us) client_negotiation.cc:769] Sending connection context
0925 02:49:39.740520 (+ 146us) client_negotiation.cc:240] Negotiation successful
0925 02:49:39.740792 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 02:49:39.741828 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.727727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48432 (local address 127.5.56.189:39407)
0925 02:49:39.727920 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.727925 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.727945 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:49:39.728084 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.728090 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.728160 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.728275 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.728946 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.729948 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.731292 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.731981 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.732125 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.735346 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.735351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.735354 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.735385 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.738338 (+ 2953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.739151 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.739155 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.739158 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.739291 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.739716 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.739720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.739724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.740055 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.740216 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.741410 (+ 1194us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.741582 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:49:39.742828 28230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
I20260925 02:49:39.756162 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.743752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.5.56.188:35383)
0925 02:49:39.743927 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.743932 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.743946 (+ 14us) server_negotiation.cc:407] Connection header received
0925 02:49:39.743988 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.743991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.744036 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.744130 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.744798 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.745439 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.746551 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.746953 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.747076 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.750500 (+ 3424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.750508 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.750513 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.750548 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.753458 (+ 2910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.754049 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.754055 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.754059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.754175 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.754615 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.754619 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.754621 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.754915 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.755025 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.755470 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.755747 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":58}
I20260925 02:49:39.756235 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.743542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38700)
0925 02:49:39.743707 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.743725 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:39.743833 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:39.744146 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:39.744150 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:39.744535 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:39.744661 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.744669 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.745556 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.745564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.746374 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.746432 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.747117 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.747123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.747371 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.747901 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:39.747918 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:39.750275 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:39.753581 (+ 3306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.753586 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.753590 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.753934 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.754297 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.754301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.754305 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.754503 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.755040 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:39.755045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:39.755229 (+ 184us) client_negotiation.cc:769] Sending connection context
0925 02:49:39.755389 (+ 160us) client_negotiation.cc:240] Negotiation successful
0925 02:49:39.755882 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":174}
I20260925 02:49:39.757192 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.742350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48444 (local address 127.5.56.189:39407)
0925 02:49:39.742527 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.742532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.742657 (+ 125us) server_negotiation.cc:407] Connection header received
0925 02:49:39.742789 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.742792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.742846 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.742917 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.744345 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.745063 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.746268 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.746755 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.746901 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.751506 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.751511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.751514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.751545 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.754462 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.755167 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.755170 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.755173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.755298 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.755707 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.755713 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.755716 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.755984 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.756299 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.756589 (+ 290us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.756851 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 02:49:39.761866 28230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b: Bootstrap starting.
I20260925 02:49:39.762363 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:39.765316 28230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:39.765666 28081 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } attempt: 4
I20260925 02:49:39.766697 28230 log.cc:824] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:39.764988 28055 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54756
I20260925 02:49:39.769712 28230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b: No bootstrap required, opened a new log
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304578, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
I20260925 02:49:39.778671 28230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } }
I20260925 02:49:39.779182 28230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c0ac2a8024944c0bd02ec4c1712cc6b, State: Initialized, Role: FOLLOWER
I20260925 02:49:39.779762 28230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } }
I20260925 02:49:39.780920 28238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } } }
I20260925 02:49:39.781333 28238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:39.782130 28230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:39.784375 28085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.766334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38710)
0925 02:49:39.766663 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.766683 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:39.766806 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:39.769219 (+ 2413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:39.769244 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:39.769774 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:39.769993 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.770008 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.771462 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.771467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.772937 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.772953 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.773959 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.773965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.774167 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.774860 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:39.774886 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:39.777387 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:39.781541 (+ 4154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.781546 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.781551 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.782056 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.782539 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.782544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.782547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.782734 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.783410 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:39.783417 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:39.783599 (+ 182us) client_negotiation.cc:769] Sending connection context
0925 02:49:39.783780 (+ 181us) client_negotiation.cc:240] Negotiation successful
0925 02:49:39.784107 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192}
I20260925 02:49:39.784732 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.766417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38710 (local address 127.5.56.188:35383)
0925 02:49:39.768884 (+ 2467us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.768891 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.768911 (+ 20us) server_negotiation.cc:407] Connection header received
0925 02:49:39.768968 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.768975 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.769051 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.769159 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.770164 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.771302 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.773090 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.773809 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.773933 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.777585 (+ 3652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.777592 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.777597 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.777633 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.781339 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.782181 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.782190 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.782195 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.782365 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.782863 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.782869 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.782872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.783278 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.783441 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.783913 (+ 472us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.784208 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2292}
I20260925 02:49:39.791980 28081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b: Bootstrap starting.
I20260925 02:49:39.796121 28081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:39.797500 28081 log.cc:824] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:39.801098 28081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b: No bootstrap required, opened a new log
W20260925 02:49:39.802088 28249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:49:39.802317 28249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:49:39.796437 28162 master_runner.cc:186] Error getting master registration for 127.5.56.190:34775: OK, instance_id { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" instance_seqno: 1790304578621914 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:49:39.808041 28162 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:49:39.808916 28081 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } }
I20260925 02:49:39.809336 28081 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a4ebf99b56149dab91d0dbbd4f95f5b, State: Initialized, Role: FOLLOWER
I20260925 02:49:39.809828 28081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } }
I20260925 02:49:39.810761 28252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } } }
I20260925 02:49:39.811162 28252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:39.812024 28081 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:39.812124 28055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:49:39.812310 28055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:39.816438 28055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } }
I20260925 02:49:39.818934 28055 leader_election.cc:288] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [CANDIDATE]: Term 1 election: Requested vote from peers 5917a13320a5485cb05263bd4cb65e21 (127.5.56.189:39407), 9c0ac2a8024944c0bd02ec4c1712cc6b (127.5.56.188:35383)
I20260925 02:49:39.819810 28204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b"
I20260925 02:49:39.820369 28204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:39.824023 28204 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ebf99b56149dab91d0dbbd4f95f5b in term 1.
I20260925 02:49:39.824714 28032 leader_election.cc:302] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a4ebf99b56149dab91d0dbbd4f95f5b, 9c0ac2a8024944c0bd02ec4c1712cc6b; no voters:
I20260925 02:49:39.825271 28252 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:39.837365 28252 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [term 1 LEADER]: Becoming Leader. State: Replica: 8a4ebf99b56149dab91d0dbbd4f95f5b, State: Running, Role: LEADER
I20260925 02:49:39.838761 28252 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } }
I20260925 02:49:39.846868 28261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a4ebf99b56149dab91d0dbbd4f95f5b. Latest consensus state: current_term: 1 leader_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } } }
I20260925 02:49:39.847416 28261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:39.849643 28271 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:39.861876 28271 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260925 02:49:39.862303 28034 tablet.cc:2369] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:49:39.871176 28155 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } attempt: 5
I20260925 02:49:39.882676 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.841725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48452 (local address 127.5.56.189:39407)
0925 02:49:39.841941 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.841948 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.864314 (+ 22366us) server_negotiation.cc:407] Connection header received
0925 02:49:39.865235 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.865243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.865338 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.865453 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.868013 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.869207 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.871078 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.871817 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.871916 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.875303 (+ 3387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.875310 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.875315 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.875372 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.878980 (+ 3608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.879837 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.879844 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.879848 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.880025 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.880480 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.880488 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.880491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.880919 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.881177 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.881994 (+ 817us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.882238 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:49:39.889115 28204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [term 1 FOLLOWER]: Refusing update from remote peer 8a4ebf99b56149dab91d0dbbd4f95f5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:49:39.890218 28252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:49:39.858027 28013 master_runner.cc:186] Error getting master registration for 127.5.56.189:39407: OK, instance_id { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" instance_seqno: 1790304579140677 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:49:39.893285 28013 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:49:39.896677 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.842205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54770 (local address 127.5.56.190:34775)
0925 02:49:39.842429 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.842436 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.854673 (+ 12237us) server_negotiation.cc:407] Connection header received
0925 02:49:39.855040 (+ 367us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.855049 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.855153 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.855247 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.862577 (+ 7330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.864203 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.869776 (+ 5573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.870563 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.870665 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.885836 (+ 15171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.885843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.885847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.885890 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.890172 (+ 4282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.893386 (+ 3214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.893396 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.893399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.893578 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.894113 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.894119 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.894124 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.894539 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.894815 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.895501 (+ 686us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.895931 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 02:49:39.898805 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.854795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.5.56.188:35383)
0925 02:49:39.854973 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.854979 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.867123 (+ 12144us) server_negotiation.cc:407] Connection header received
0925 02:49:39.867404 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.867410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.867498 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.867580 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.878533 (+ 10953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.879696 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.884175 (+ 4479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.884780 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.884883 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.889145 (+ 4262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.889153 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.889157 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.889198 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.892669 (+ 3471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.896380 (+ 3711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.896388 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.896391 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.896544 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.897030 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.897037 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.897042 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.897399 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.897638 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.898168 (+ 530us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.898428 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
Sep 25 02:49:39 dist-test-slave-n563 krb5kdc[28000](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304579, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
I20260925 02:49:39.908499 28238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a4ebf99b56149dab91d0dbbd4f95f5b. Latest consensus state: current_term: 1 leader_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } } }
I20260925 02:49:39.908802 28238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:39.919921 28261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } } }
I20260925 02:49:39.920221 28261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:39.924434 28157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.871842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:35383 (local address 127.0.0.1:38742)
0925 02:49:39.872199 (+ 357us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:39.872218 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:39.872331 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:39.899401 (+ 27070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:39.899408 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:39.899923 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:39.900162 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.900181 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.901722 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.901728 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.903210 (+ 1482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:39.903227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.904410 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.904419 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:39.904635 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.905296 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:39.905319 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:39.908576 (+ 3257us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:39.914337 (+ 5761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.914342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.914346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.915046 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.922504 (+ 7458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:39.922510 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:39.922515 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:39.922713 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:39.923414 (+ 701us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:39.923419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:39.923581 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 02:49:39.923717 (+ 136us) client_negotiation.cc:240] Negotiation successful
0925 02:49:39.924048 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238}
I20260925 02:49:39.924583 28238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } } }
I20260925 02:49:39.924840 28238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:39.925670 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:39.872551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38742 (local address 127.5.56.188:35383)
0925 02:49:39.899098 (+ 26547us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:39.899105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:39.899122 (+ 17us) server_negotiation.cc:407] Connection header received
0925 02:49:39.899168 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:39.899175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:39.899245 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:39.899342 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:39.900436 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.901525 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.903355 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:39.904083 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:39.904207 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:39.908852 (+ 4645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:39.908862 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:39.908867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:39.908932 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:39.914191 (+ 5259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.915222 (+ 1031us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.915227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.915230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.915384 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:39.922877 (+ 7493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:39.922883 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:39.922886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:39.923259 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:39.924859 (+ 1600us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:39.924987 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 02:49:39.925253 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26443,"thread_start_us":34079,"threads_started":1}
I20260925 02:49:39.932561 28155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21: Bootstrap starting.
I20260925 02:49:39.932920 28271 catalog_manager.cc:1347] Generated new cluster ID: 702a1df7eaa649ac86e20019545cf236
I20260925 02:49:39.933053 28271 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:39.940076 28155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:39.941463 28155 log.cc:824] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:39.958824 28155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21: No bootstrap required, opened a new log
I20260925 02:49:39.961330 28271 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:39.965615 28271 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:39.976233 28155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } }
I20260925 02:49:39.976804 28155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5917a13320a5485cb05263bd4cb65e21, State: Initialized, Role: FOLLOWER
I20260925 02:49:39.977412 28155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } }
I20260925 02:49:39.989231 28271 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b: Generated new TSK 0
I20260925 02:49:39.990465 28271 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:39.992699 28155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:39.993093 28129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:39.997509 28129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [term 1 FOLLOWER]: Refusing update from remote peer 8a4ebf99b56149dab91d0dbbd4f95f5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:49:39.998746 28252 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:49:39.999258 28130 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5917a13320a5485cb05263bd4cb65e21"
I20260925 02:49:40.002727 28130 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a4ebf99b56149dab91d0dbbd4f95f5b in term 1.
I20260925 02:49:40.007830 28282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } } }
I20260925 02:49:40.008478 28282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:40.038882 28287 mvcc.cc:205] Tried to move back new op lower bound from 7333087559213981696 to 7333087559041232896. Current Snapshot: MvccSnapshot[applied={T|T < 7333087559213981696}]
I20260925 02:49:40.048128 28285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } } }
I20260925 02:49:40.048691 28285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:40.049109 28282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a4ebf99b56149dab91d0dbbd4f95f5b. Latest consensus state: current_term: 1 leader_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } } }
I20260925 02:49:40.049475 28282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:40.077684 28299 catalog_manager.cc:1259] Loaded cluster ID: 702a1df7eaa649ac86e20019545cf236
I20260925 02:49:40.077900 28299 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21: loading cluster ID for follower catalog manager: success
I20260925 02:49:40.081844 28299 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21: acquiring CA information for follower catalog manager: success
I20260925 02:49:40.084730 28299 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5917a13320a5485cb05263bd4cb65e21: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:49:40.121871 5346 mini_kdc.cc:348] Time spent kdestroy: real 0.114s user 0.003s sys 0.021s
W20260925 02:49:40.168121 28159 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:40.155515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.5.56.189:39407)
0925 02:49:40.155804 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.155812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.165934 (+ 10122us) server_negotiation.cc:407] Connection header received
0925 02:49:40.166476 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.166482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.166593 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.166674 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:40.167874 (+ 1200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48454: BlockingRecv error: recv got EOF from 127.0.0.1:48454 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260925 02:49:40.171406 28235 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:40.162025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54772 (local address 127.5.56.190:34775)
0925 02:49:40.162257 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.162263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.169087 (+ 6824us) server_negotiation.cc:407] Connection header received
0925 02:49:40.169301 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.169307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.169430 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.169528 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:40.171119 (+ 1591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54772: BlockingRecv error: recv got EOF from 127.0.0.1:54772 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260925 02:49:40.174744 28233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:40.172512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38750 (local address 127.5.56.188:35383)
0925 02:49:40.172731 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.172739 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.173163 (+ 424us) server_negotiation.cc:407] Connection header received
0925 02:49:40.173401 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.173409 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.173488 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.173560 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:40.174491 (+ 931us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38750: BlockingRecv error: recv got EOF from 127.0.0.1:38750 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:49:40.229301 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.209616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48462 (local address 127.5.56.189:39407)
0925 02:49:40.209824 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.209831 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.220472 (+ 10641us) server_negotiation.cc:407] Connection header received
0925 02:49:40.221057 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.221063 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.221254 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.221406 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.222125 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.223426 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.225478 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.226398 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.226500 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.227664 (+ 1164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.228414 (+ 750us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.228550 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.228809 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.228908 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260925 02:49:40.269060 28233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:40.259499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.5.56.188:35383)
0925 02:49:40.259735 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.259742 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.266805 (+ 7063us) server_negotiation.cc:407] Connection header received
0925 02:49:40.267378 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.267385 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.267532 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.267617 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:40.268814 (+ 1197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38760: BlockingRecv error: recv got EOF from 127.0.0.1:38760 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:49:40.319317 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.300934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48470 (local address 127.5.56.189:39407)
0925 02:49:40.301123 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.301141 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.311929 (+ 10788us) server_negotiation.cc:407] Connection header received
0925 02:49:40.312498 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.312505 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.312606 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.312687 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.313400 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.314672 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.316876 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.317630 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.317731 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.318402 (+ 671us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.318598 (+ 196us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.318662 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.318899 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.318993 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
W20260925 02:49:40.362850 28233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:40.353497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.5.56.188:35383)
0925 02:49:40.353727 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.353734 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.360763 (+ 7029us) server_negotiation.cc:407] Connection header received
0925 02:49:40.361272 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.361281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.361382 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.361465 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:40.362599 (+ 1134us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38774: BlockingRecv error: recv got EOF from 127.0.0.1:38774 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
I20260925 02:49:40.395521 28252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a4ebf99b56149dab91d0dbbd4f95f5b" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 34775 } } peers { permanent_uuid: "5917a13320a5485cb05263bd4cb65e21" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 39407 } } peers { permanent_uuid: "9c0ac2a8024944c0bd02ec4c1712cc6b" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 35383 } } }
I20260925 02:49:40.395952 28252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a4ebf99b56149dab91d0dbbd4f95f5b [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:40.415235 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.395794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48478 (local address 127.5.56.189:39407)
0925 02:49:40.395997 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.396004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.408013 (+ 12009us) server_negotiation.cc:407] Connection header received
0925 02:49:40.408552 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.408560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.408653 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.408718 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.409505 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.410734 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.412802 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.413510 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.413615 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.414262 (+ 647us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.414435 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.414499 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.414733 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.414837 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260925 02:49:40.466042 28233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:40.453005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.5.56.188:35383)
0925 02:49:40.453208 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.453215 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.464100 (+ 10885us) server_negotiation.cc:407] Connection header received
0925 02:49:40.464425 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.464432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.464549 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.464628 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:40.465757 (+ 1129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38776: BlockingRecv error: recv got EOF from 127.0.0.1:38776 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20260925 02:49:40.525658 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.507647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48488 (local address 127.5.56.189:39407)
0925 02:49:40.507853 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.507859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.518626 (+ 10767us) server_negotiation.cc:407] Connection header received
0925 02:49:40.519173 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.519181 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.519285 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.519352 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.520108 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.521284 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.523388 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.524076 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.524176 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.524824 (+ 648us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.524974 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.525038 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.525270 (+ 232us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.525365 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 02:49:40.564668 28233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:40.557416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38790 (local address 127.5.56.188:35383)
0925 02:49:40.557704 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.557713 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.563062 (+ 5349us) server_negotiation.cc:407] Connection header received
0925 02:49:40.563271 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.563277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.563396 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.563538 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:40.564445 (+ 907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38790: BlockingRecv error: recv got EOF from 127.0.0.1:38790 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
I20260925 02:49:40.582751 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.576260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48490 (local address 127.5.56.189:39407)
0925 02:49:40.576484 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.576490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.576860 (+ 370us) server_negotiation.cc:407] Connection header received
0925 02:49:40.577068 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.577072 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.577144 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.577230 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.577690 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.578563 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.580615 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.581120 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.581224 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.581814 (+ 590us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.582015 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.582092 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.582293 (+ 201us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.582372 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260925 02:49:40.591624 28233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:40.589463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38794 (local address 127.5.56.188:35383)
0925 02:49:40.589697 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.589703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.590059 (+ 356us) server_negotiation.cc:407] Connection header received
0925 02:49:40.590241 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.590245 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.590316 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.590482 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:40.591331 (+ 849us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38794: BlockingRecv error: recv got EOF from 127.0.0.1:38794 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260925 02:49:40.617808 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.611902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48494 (local address 127.5.56.189:39407)
0925 02:49:40.612138 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.612144 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.612494 (+ 350us) server_negotiation.cc:407] Connection header received
0925 02:49:40.612752 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.612756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.612828 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.612923 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.613511 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.614316 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.616012 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.616461 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.616621 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.617079 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.617206 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.617305 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.617459 (+ 154us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.617539 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
W20260925 02:49:40.625612 28233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:40.623624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.5.56.188:35383)
0925 02:49:40.623807 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.623812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.624253 (+ 441us) server_negotiation.cc:407] Connection header received
0925 02:49:40.624464 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.624476 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.624549 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.624656 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:40.625371 (+ 715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38796: BlockingRecv error: recv got EOF from 127.0.0.1:38796 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 02:49:40.668200 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.661709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48498 (local address 127.5.56.189:39407)
0925 02:49:40.662004 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.662011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.662439 (+ 428us) server_negotiation.cc:407] Connection header received
0925 02:49:40.662705 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.662714 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.662846 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.662973 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.663435 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.664611 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.665985 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.666740 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.666882 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.667339 (+ 457us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.667485 (+ 146us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.667577 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.667775 (+ 198us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.667870 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
W20260925 02:49:40.683435 28233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:40.681463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.5.56.188:35383)
0925 02:49:40.681667 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.681674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.682064 (+ 390us) server_negotiation.cc:407] Connection header received
0925 02:49:40.682217 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.682224 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.682331 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.682488 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:40.683191 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38802: BlockingRecv error: recv got EOF from 127.0.0.1:38802 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260925 02:49:40.756490 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.751262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48510 (local address 127.5.56.189:39407)
0925 02:49:40.751504 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.751512 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.751763 (+ 251us) server_negotiation.cc:407] Connection header received
0925 02:49:40.751943 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.751948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.752020 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.752101 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.752528 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.753260 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.754688 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.755109 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.755207 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.755704 (+ 497us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.755849 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.755944 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.756109 (+ 165us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.756177 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260925 02:49:40.764549 28233 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:40.762620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.5.56.188:35383)
0925 02:49:40.762878 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.762885 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.763158 (+ 273us) server_negotiation.cc:407] Connection header received
0925 02:49:40.763385 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.763391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.763473 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.763578 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:40.764347 (+ 769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38808: BlockingRecv error: recv got EOF from 127.0.0.1:38808 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260925 02:49:40.804797 28249 catalog_manager.cc:1259] Loaded cluster ID: 702a1df7eaa649ac86e20019545cf236
I20260925 02:49:40.804927 28249 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b: loading cluster ID for follower catalog manager: success
I20260925 02:49:40.807547 28249 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b: acquiring CA information for follower catalog manager: success
I20260925 02:49:40.809360 28249 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c0ac2a8024944c0bd02ec4c1712cc6b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:49:40.901636 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.896421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48520 (local address 127.5.56.189:39407)
0925 02:49:40.896642 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.896648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.897060 (+ 412us) server_negotiation.cc:407] Connection header received
0925 02:49:40.897230 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.897235 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.897306 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.897392 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.897810 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.898598 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.899947 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.900366 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.900506 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.900899 (+ 393us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.901004 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.901103 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.901255 (+ 152us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.901346 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260925 02:49:40.913317 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.907416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.5.56.188:35383)
0925 02:49:40.907614 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.907620 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.907943 (+ 323us) server_negotiation.cc:407] Connection header received
0925 02:49:40.908155 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.908160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.908319 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.908442 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.908914 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.909762 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.911096 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.911512 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.911606 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.912114 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.912676 (+ 562us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.912785 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.912914 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.912984 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 02:49:40.928198 28159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.919913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48536 (local address 127.5.56.189:39407)
0925 02:49:40.920102 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.920110 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.923010 (+ 2900us) server_negotiation.cc:407] Connection header received
0925 02:49:40.923204 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.923210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.923304 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.923402 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.923905 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.924651 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.926178 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.926810 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.926959 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.927478 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.927612 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.927703 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.927841 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.927936 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 02:49:40.933974 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.919698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.5.56.190:34775)
0925 02:49:40.927301 (+ 7603us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.927305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.927321 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:49:40.927459 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.927465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.927723 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.927948 (+ 225us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.928610 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.929594 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.931711 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.932277 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.932403 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.932872 (+ 469us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.933322 (+ 450us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.933429 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.933577 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.933668 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7499,"thread_start_us":7401,"threads_started":1}
I20260925 02:49:40.934073 28233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:40.920702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.5.56.188:35383)
0925 02:49:40.920924 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:40.920933 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:40.926969 (+ 6036us) server_negotiation.cc:407] Connection header received
0925 02:49:40.928365 (+ 1396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:40.928374 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:40.928477 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:40.928588 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:40.929210 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.930122 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.932024 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:40.932702 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:40.932850 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:40.933229 (+ 379us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:40.933351 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:40.933429 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:40.933677 (+ 248us) server_negotiation.cc:299] Negotiation successful
0925 02:49:40.933758 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 02:49:40.938881 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 28013
I20260925 02:49:40.940447 28076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:41.051397 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 28013
W20260925 02:49:41.072750 28107 connection.cc:535] server connection from 127.0.0.1:48430 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:49:41.072958 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 28088
I20260925 02:49:41.074046 28150 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:41.181151 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 28088
I20260925 02:49:41.202564 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 28162
I20260925 02:49:41.203794 28225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:41.315263 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 28162
2026-09-25T02:49:41Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4527 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 02:49:41 dist-test-slave-n563 krb5kdc[28413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 02:49:41 dist-test-slave-n563 krb5kdc[28413](info): set up 2 sockets
Sep 25 02:49:41 dist-test-slave-n563 krb5kdc[28413](info): commencing operation
krb5kdc: starting...
W20260925 02:49:42.431986 5346 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/test-user.keytab.
Sep 25 02:49:42 dist-test-slave-n563 krb5kdc[28413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T02:49:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T02:49:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.5.56.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
Entry for principal kudu/127.5.56.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab.
I20260925 02:49:42.708097 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:35297
--webserver_interface=127.5.56.190
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:35297,127.5.56.189:44419,127.5.56.188:36105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 02:49:43.048831 28426 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:49:43.049158 28426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:43.049237 28426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:43.049325 28426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:43.056645 28426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:43.056751 28426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:43.056810 28426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:43.056861 28426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:43.056967 28426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:43.067698 28426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:35297,127.5.56.189:44419,127.5.56.188:36105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.190.keytab
--principal=kudu/127.5.56.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.190:35297
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.56.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:43.069010 28426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:43.070327 28426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:43.082620 28431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:43.082805 28434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:43.082872 28432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:43.083755 28426 server_base.cc:1034] running on GCE node
I20260925 02:49:43.084522 28426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:43.085943 28426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:43.087220 28426 hybrid_clock.cc:648] HybridClock initialized: now 1790304583087154 us; error 38 us; skew 500 ppm
I20260925 02:49:43.087554 28426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:43 dist-test-slave-n563 krb5kdc[28413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304583, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:43.090744 28426 init.cc:377] Logged in from keytab as kudu/127.5.56.190@KRBTEST.COM (short username kudu)
I20260925 02:49:43.092252 28426 webserver.cc:460] Webserver started at http://127.5.56.190:43733/ using document root <none> and password file <none>
I20260925 02:49:43.092957 28426 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:43.093091 28426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:43.093427 28426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:43.096566 28426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/data/instance:
uuid: "1b588414e483452cb7b115e0d2f214bf"
format_stamp: "Formatted at 2026-09-25 02:49:43 on dist-test-slave-n563"
server_key: "c0dd5964d3a7443cba472d56b0e35512"
server_key_iv: "23dab89978c5313ecbe11cf0112da7f7"
server_key_version: "encryptionkey@0"
I20260925 02:49:43.097397 28426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/wal/instance:
uuid: "1b588414e483452cb7b115e0d2f214bf"
format_stamp: "Formatted at 2026-09-25 02:49:43 on dist-test-slave-n563"
server_key: "c0dd5964d3a7443cba472d56b0e35512"
server_key_iv: "23dab89978c5313ecbe11cf0112da7f7"
server_key_version: "encryptionkey@0"
I20260925 02:49:43.102871 28426 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260925 02:49:43.106302 28441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:43.107826 28426 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:49:43.108042 28426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/wal
uuid: "1b588414e483452cb7b115e0d2f214bf"
format_stamp: "Formatted at 2026-09-25 02:49:43 on dist-test-slave-n563"
server_key: "c0dd5964d3a7443cba472d56b0e35512"
server_key_iv: "23dab89978c5313ecbe11cf0112da7f7"
server_key_version: "encryptionkey@0"
I20260925 02:49:43.108300 28426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:43.128897 28426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:43.133129 28426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:43.133482 28426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:43.152694 28426 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.190:35297
I20260925 02:49:43.152732 28492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.190:35297 every 8 connection(s)
I20260925 02:49:43.154088 28426 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/data/info.pb
I20260925 02:49:43.154878 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 28426
I20260925 02:49:43.155072 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-0/wal/instance
I20260925 02:49:43.155646 5346 external_mini_cluster.cc:1428] Setting key eaf7734ef98d6e16906d077c9ac97f38
I20260925 02:49:43.159039 28494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:43.168751 28494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.5.56.189@KRBTEST.COM; defaulting to no policy
Sep 25 02:49:43 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304583, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.189@KRBTEST.COM" created.
I20260925 02:49:43.194515 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.172278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35992 (local address 127.5.56.190:35297)
0925 02:49:43.174322 (+ 2044us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:43.174327 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:43.174372 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:49:43.176717 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:43.176723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:43.176974 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:43.177134 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:43.178840 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.180021 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.182318 (+ 2297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.182985 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.183124 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:43.187370 (+ 4246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:43.187378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:43.187392 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:43.187421 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:43.190203 (+ 2782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:43.191138 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:43.191144 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:43.191146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:43.191287 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:43.191751 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:43.191754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:43.191756 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:43.192259 (+ 503us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:43.192419 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:43.193074 (+ 655us) server_negotiation.cc:299] Negotiation successful
0925 02:49:43.193384 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":1858,"thread_start_us":1598,"threads_started":1}
I20260925 02:49:43.194562 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.172278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35297 (local address 127.0.0.1:35992)
0925 02:49:43.173313 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.173372 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:43.176289 (+ 2917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:43.177223 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:43.177233 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:43.178082 (+ 849us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:43.178628 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:43.178655 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.180182 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.180191 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:43.182167 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:43.182186 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.183251 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.183263 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:43.183498 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:43.184205 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:43.184237 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:43.187106 (+ 2869us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:43.190349 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:43.190366 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:43.190404 (+ 38us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:43.191003 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:43.191410 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:43.191418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:43.191424 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:43.191635 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:43.192409 (+ 774us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:43.192420 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:43.192678 (+ 258us) client_negotiation.cc:769] Sending connection context
0925 02:49:43.192948 (+ 270us) client_negotiation.cc:240] Negotiation successful
0925 02:49:43.193570 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"mutex_wait_us":74,"thread_start_us":455,"threads_started":1}
I20260925 02:49:43.197412 28494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:49:43.198519 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.198071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42640)
0925 02:49:43.198225 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.198342 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 02:49:43.198767 28444 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
I20260925 02:49:43.200313 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.199845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42656)
0925 02:49:43.200029 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.200124 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 02:49:43.200521 28494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:44419: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab.
Entry for principal kudu/127.5.56.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab.
I20260925 02:49:43.223688 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.189:44419
--webserver_interface=127.5.56.189
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab
--principal=kudu/127.5.56.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:35297,127.5.56.189:44419,127.5.56.188:36105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:49:43.249898 28494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } attempt: 1
I20260925 02:49:43.251245 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.250707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42670)
0925 02:49:43.250933 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.251074 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260925 02:49:43.252346 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.251925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42676)
0925 02:49:43.252074 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.252154 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260925 02:49:43.252550 28494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:44419: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
I20260925 02:49:43.320823 28494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } attempt: 2
I20260925 02:49:43.322079 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.321536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42692)
0925 02:49:43.321795 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.321909 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260925 02:49:43.323314 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.322875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42702)
0925 02:49:43.323050 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.323147 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 02:49:43.323524 28494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:44419: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
I20260925 02:49:43.414805 28494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } attempt: 3
I20260925 02:49:43.416252 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.415597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42706)
0925 02:49:43.415869 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.416007 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 02:49:43.417382 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.416949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42710)
0925 02:49:43.417109 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.417208 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260925 02:49:43.417580 28494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:44419: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
I20260925 02:49:43.560835 28494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } attempt: 4
I20260925 02:49:43.562139 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.561529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42714)
0925 02:49:43.561855 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.561963 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260925 02:49:43.563257 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.562847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42724)
0925 02:49:43.563009 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.563107 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 02:49:43.563464 28494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.189:44419: Network error: Client connection negotiation failed: client connection to 127.5.56.189:44419: connect: Connection refused (error 111)
W20260925 02:49:43.565949 28501 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:49:43.566275 28501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:43.566347 28501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:43.566501 28501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 02:49:43.574151 28501 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:43.574236 28501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:43.574293 28501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:43.574343 28501 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:43.574445 28501 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:43.584540 28501 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:35297,127.5.56.189:44419,127.5.56.188:36105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.189.keytab
--principal=kudu/127.5.56.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.189:44419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.5.56.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:43.585842 28501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:43.587235 28501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 02:49:43.599216 28507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:43.599273 28509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:43.599186 28506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:43.600080 28501 server_base.cc:1034] running on GCE node
I20260925 02:49:43.600922 28501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:43.602434 28501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:43.603689 28501 hybrid_clock.cc:648] HybridClock initialized: now 1790304583603607 us; error 60 us; skew 500 ppm
I20260925 02:49:43.604035 28501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:43 dist-test-slave-n563 krb5kdc[28413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304583, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:43.607400 28501 init.cc:377] Logged in from keytab as kudu/127.5.56.189@KRBTEST.COM (short username kudu)
I20260925 02:49:43.608925 28501 webserver.cc:460] Webserver started at http://127.5.56.189:46033/ using document root <none> and password file <none>
I20260925 02:49:43.609599 28501 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:43.609687 28501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:43.609959 28501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:43.612952 28501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/data/instance:
uuid: "e979d7e9a61a4c1fab4f123145fe8f28"
format_stamp: "Formatted at 2026-09-25 02:49:43 on dist-test-slave-n563"
server_key: "9d7ada01fb41784b77f50302179a3b21"
server_key_iv: "c99251195ace6faf95082ad984968030"
server_key_version: "encryptionkey@0"
I20260925 02:49:43.613664 28501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/wal/instance:
uuid: "e979d7e9a61a4c1fab4f123145fe8f28"
format_stamp: "Formatted at 2026-09-25 02:49:43 on dist-test-slave-n563"
server_key: "9d7ada01fb41784b77f50302179a3b21"
server_key_iv: "c99251195ace6faf95082ad984968030"
server_key_version: "encryptionkey@0"
I20260925 02:49:43.618907 28501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260925 02:49:43.623188 28516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:43.624523 28501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260925 02:49:43.624704 28501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/wal
uuid: "e979d7e9a61a4c1fab4f123145fe8f28"
format_stamp: "Formatted at 2026-09-25 02:49:43 on dist-test-slave-n563"
server_key: "9d7ada01fb41784b77f50302179a3b21"
server_key_iv: "c99251195ace6faf95082ad984968030"
server_key_version: "encryptionkey@0"
I20260925 02:49:43.624934 28501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:43.649825 28501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:43.653843 28501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:43.654127 28501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:43.672963 28501 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.189:44419
I20260925 02:49:43.672945 28567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.189:44419 every 8 connection(s)
I20260925 02:49:43.674369 28501 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/data/info.pb
I20260925 02:49:43.678149 28568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:43.681757 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 28501
I20260925 02:49:43.681972 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-1/wal/instance
I20260925 02:49:43.682451 5346 external_mini_cluster.cc:1428] Setting key b750f02bd16b52615ddf29283db0110b
I20260925 02:49:43.684746 28568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:49:43 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304583, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:49:43.706360 28498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.687305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36004 (local address 127.5.56.190:35297)
0925 02:49:43.687547 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:43.687556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:43.688794 (+ 1238us) server_negotiation.cc:407] Connection header received
0925 02:49:43.691603 (+ 2809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:43.691612 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:43.691713 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:43.691861 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:43.693374 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.694061 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.695624 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.696048 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.696180 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:43.699626 (+ 3446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:43.699634 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:43.699642 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:43.699688 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:43.703331 (+ 3643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:43.704052 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:43.704065 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:43.704069 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:43.704186 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:43.704603 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:43.704608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:43.704613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:43.704897 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:43.705054 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:43.705792 (+ 738us) server_negotiation.cc:299] Negotiation successful
0925 02:49:43.706023 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:49:43.707432 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.687779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35297 (local address 127.0.0.1:36004)
0925 02:49:43.688588 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.688639 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:43.691327 (+ 2688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:43.692056 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:43.692064 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:43.692597 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:43.693207 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:43.693226 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.694230 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.694235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:43.695469 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:43.695481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.696285 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.696292 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:43.696517 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:43.697195 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:43.697218 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:43.699395 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:43.703466 (+ 4071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:43.703473 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:43.703487 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:43.703917 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:43.704300 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:43.704305 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:43.704308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:43.704481 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:43.705028 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:43.705035 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:43.705417 (+ 382us) client_negotiation.cc:769] Sending connection context
0925 02:49:43.705688 (+ 271us) client_negotiation.cc:240] Negotiation successful
0925 02:49:43.706206 (+ 518us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":308,"threads_started":1}
WARNING: no policy specified for kudu/127.5.56.188@KRBTEST.COM; defaulting to no policy
I20260925 02:49:43.709316 28568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.5.56.188@KRBTEST.COM" created.
Sep 25 02:49:43 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304583, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
I20260925 02:49:43.725589 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.710301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.5.56.189:44419)
0925 02:49:43.711062 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:43.711071 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:43.711106 (+ 35us) server_negotiation.cc:407] Connection header received
0925 02:49:43.711160 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:43.711169 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:43.711425 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:43.711573 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:43.712386 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.714141 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.715326 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.716237 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.716379 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:43.719428 (+ 3049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:43.719444 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:43.719464 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:43.719511 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:43.722587 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:43.723428 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:43.723434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:43.723436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:43.723550 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:43.723968 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:43.723971 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:43.723973 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:43.724344 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:43.724492 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:43.724955 (+ 463us) server_negotiation.cc:299] Negotiation successful
0925 02:49:43.725257 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":120,"server-negotiator.queue_time_us":583,"thread_start_us":431,"threads_started":1}
I20260925 02:49:43.725631 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.710003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42732)
0925 02:49:43.710195 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.710209 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:43.710296 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:43.711588 (+ 1292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:43.711591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:43.712064 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:43.712204 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:43.712212 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.714274 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.714277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:43.715181 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:43.715192 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.716388 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.716391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:43.716546 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:43.717142 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:43.717159 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:43.719213 (+ 2054us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:43.722761 (+ 3548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:43.722769 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:43.722776 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:43.723295 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:43.723659 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:43.723667 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:43.723673 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:43.723858 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:43.724475 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:43.724483 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:43.724650 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 02:49:43.724794 (+ 144us) client_negotiation.cc:240] Negotiation successful
0925 02:49:43.725123 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":2}
I20260925 02:49:43.728188 28568 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:49:43.729486 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.728828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52514)
0925 02:49:43.729086 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.729257 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 02:49:43.729811 28519 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
I20260925 02:49:43.731587 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.731135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52528)
0925 02:49:43.731323 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.731439 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 02:49:43.731827 28568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:36105: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.5.56.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab.
Entry for principal kudu/127.5.56.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab.
I20260925 02:49:43.746721 5346 external_mini_cluster.cc:1326] Running /tmp/dist-test-task87C_4R/build/asan/bin/kudu
/tmp/dist-test-task87C_4R/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.188:36105
--webserver_interface=127.5.56.188
--webserver_port=0
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab
--principal=kudu/127.5.56.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.5.56.148:43389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.56.190:35297,127.5.56.189:44419,127.5.56.188:36105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 02:49:43.781193 28568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } attempt: 1
I20260925 02:49:43.782583 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.781992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52536)
0925 02:49:43.782232 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.782395 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260925 02:49:43.783643 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.783256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52548)
0925 02:49:43.783432 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.783517 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260925 02:49:43.783869 28568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:36105: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
I20260925 02:49:43.852136 28568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } attempt: 2
I20260925 02:49:43.853365 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.852815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52556)
0925 02:49:43.853052 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.853163 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260925 02:49:43.854427 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.854017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52566)
0925 02:49:43.854172 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.854262 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260925 02:49:43.854701 28568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:36105: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
I20260925 02:49:43.862695 28494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } attempt: 5
Sep 25 02:49:43 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304583, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
I20260925 02:49:43.877398 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.863422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42742)
0925 02:49:43.863641 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.863662 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:43.863799 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:43.864207 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:43.864210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:43.864585 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:43.864760 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:43.864789 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.866205 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.866209 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:43.867192 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:43.867212 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.868152 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.868156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:43.868293 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:43.868900 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:43.868917 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:43.871130 (+ 2213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:43.874897 (+ 3767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:43.874901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:43.874904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:43.875293 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:43.875725 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:43.875728 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:43.875731 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:43.875855 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:43.876563 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:43.876566 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:43.876696 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 02:49:43.876843 (+ 147us) client_negotiation.cc:240] Negotiation successful
0925 02:49:43.877138 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260925 02:49:43.877605 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.863605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42742 (local address 127.5.56.189:44419)
0925 02:49:43.863833 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:43.863842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:43.863870 (+ 28us) server_negotiation.cc:407] Connection header received
0925 02:49:43.863947 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:43.863953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:43.864048 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:43.864173 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:43.864930 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.866069 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.867351 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:43.868006 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:43.868127 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:43.871334 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:43.871340 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:43.871345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:43.871384 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:43.874757 (+ 3373us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:43.875423 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:43.875430 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:43.875436 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:43.875603 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:43.875969 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:43.875975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:43.875981 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:43.876432 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:43.876609 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:43.877016 (+ 407us) server_negotiation.cc:299] Negotiation successful
0925 02:49:43.877295 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260925 02:49:43.878633 28494 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } has no permanent_uuid. Determining permanent_uuid...
I20260925 02:49:43.879629 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.879267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52580)
0925 02:49:43.879424 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.879497 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 02:49:43.880522 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.880199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52590)
0925 02:49:43.880344 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.880432 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260925 02:49:43.880736 28494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:36105: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
I20260925 02:49:43.931963 28494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } attempt: 1
I20260925 02:49:43.933161 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.932694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52606)
0925 02:49:43.932898 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.932989 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260925 02:49:43.934171 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.933833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52612)
0925 02:49:43.933984 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.934060 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 02:49:43.934433 28494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:36105: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
I20260925 02:49:43.945955 28568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } attempt: 3
I20260925 02:49:43.947363 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.946704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52618)
0925 02:49:43.946972 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.947106 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260925 02:49:43.948650 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:43.948080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52626)
0925 02:49:43.948362 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:43.948472 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155}
W20260925 02:49:43.948925 28568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:36105: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
I20260925 02:49:44.002631 28494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } attempt: 2
I20260925 02:49:44.003780 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.003282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52638)
0925 02:49:44.003529 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:44.003623 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260925 02:49:44.004860 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.004555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52650)
0925 02:49:44.004692 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:44.004759 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260925 02:49:44.005157 28494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:36105: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
W20260925 02:49:44.089170 28575 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 02:49:44.089562 28575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 02:49:44.089705 28575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 02:49:44.089831 28575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260925 02:49:44.092187 28568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } attempt: 4
I20260925 02:49:44.093506 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.092906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52662)
0925 02:49:44.093171 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:44.093297 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260925 02:49:44.094624 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.094200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52668)
0925 02:49:44.094394 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:44.094492 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260925 02:49:44.095409 28568 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:36105: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
W20260925 02:49:44.098518 28575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 02:49:44.098594 28575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 02:49:44.098711 28575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 02:49:44.098794 28575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 02:49:44.098878 28575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 02:49:44.109191 28575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.56.148:43389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.5.56.190:35297,127.5.56.189:44419,127.5.56.188:36105
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/krb5kdc/kudu_127.5.56.188.keytab
--principal=kudu/127.5.56.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.56.188:36105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.5.56.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 25 Sep 2026 01:56:34 UTC on ccycloud.quasar-vhdbox.root.comops.site
build id 4671762
ASAN enabled
I20260925 02:49:44.110682 28575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 02:49:44.111382 28494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } attempt: 3
I20260925 02:49:44.112282 28575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 02:49:44.112674 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.112145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52674)
0925 02:49:44.112397 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:44.112506 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260925 02:49:44.113867 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.113461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52678)
0925 02:49:44.113677 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:44.113751 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260925 02:49:44.114133 28494 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.5.56.188:36105: Network error: Client connection negotiation failed: client connection to 127.5.56.188:36105: connect: Connection refused (error 111)
W20260925 02:49:44.125056 28583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:44.125137 28580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 02:49:44.125084 28581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 02:49:44.127964 28575 server_base.cc:1034] running on GCE node
I20260925 02:49:44.128964 28575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 02:49:44.130883 28575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 02:49:44.132128 28575 hybrid_clock.cc:648] HybridClock initialized: now 1790304584132055 us; error 32 us; skew 500 ppm
I20260925 02:49:44.132511 28575 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 25 02:49:44 dist-test-slave-n563 krb5kdc[28413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304584, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 02:49:44.136222 28575 init.cc:377] Logged in from keytab as kudu/127.5.56.188@KRBTEST.COM (short username kudu)
I20260925 02:49:44.137789 28575 webserver.cc:460] Webserver started at http://127.5.56.188:40441/ using document root <none> and password file <none>
I20260925 02:49:44.138666 28575 fs_manager.cc:359] Metadata directory not provided
I20260925 02:49:44.138821 28575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 02:49:44.139183 28575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 02:49:44.142659 28575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/data/instance:
uuid: "0ba095a0af7a42d4961a10c8a4cda783"
format_stamp: "Formatted at 2026-09-25 02:49:44 on dist-test-slave-n563"
server_key: "a7dd7e13d2ec2bc49966b9e2e86fb640"
server_key_iv: "b7270c90dcd86817de8e08407dd5c611"
server_key_version: "encryptionkey@0"
I20260925 02:49:44.143582 28575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/wal/instance:
uuid: "0ba095a0af7a42d4961a10c8a4cda783"
format_stamp: "Formatted at 2026-09-25 02:49:44 on dist-test-slave-n563"
server_key: "a7dd7e13d2ec2bc49966b9e2e86fb640"
server_key_iv: "b7270c90dcd86817de8e08407dd5c611"
server_key_version: "encryptionkey@0"
I20260925 02:49:44.149179 28575 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260925 02:49:44.153394 28590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 02:49:44.154914 28575 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260925 02:49:44.155150 28575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/data,/tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/wal
uuid: "0ba095a0af7a42d4961a10c8a4cda783"
format_stamp: "Formatted at 2026-09-25 02:49:44 on dist-test-slave-n563"
server_key: "a7dd7e13d2ec2bc49966b9e2e86fb640"
server_key_iv: "b7270c90dcd86817de8e08407dd5c611"
server_key_version: "encryptionkey@0"
I20260925 02:49:44.155414 28575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 02:49:44.190603 28575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 02:49:44.195986 28575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 02:49:44.196331 28575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 02:49:44.217481 28575 rpc_server.cc:307] RPC server started. Bound to: 127.5.56.188:36105
I20260925 02:49:44.217502 28641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.5.56.188:36105 every 8 connection(s)
I20260925 02:49:44.219177 28575 server_base.cc:1166] Dumped server information to /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/data/info.pb
I20260925 02:49:44.223078 28642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 02:49:44.223559 5346 external_mini_cluster.cc:1388] Started /tmp/dist-test-task87C_4R/build/asan/bin/kudu as pid 28575
I20260925 02:49:44.223769 5346 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task87C_4R/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790304212743187-5346-0/minicluster-data/master-2/wal/instance
I20260925 02:49:44.224295 5346 external_mini_cluster.cc:1428] Setting key 8df75439f8c601eeb34c93c8c2459c6a
I20260925 02:49:44.232175 28642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:49:44 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
Sep 25 02:49:44 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304584, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
I20260925 02:49:44.249816 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.226456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52692 (local address 127.5.56.188:36105)
0925 02:49:44.227284 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.227298 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.227343 (+ 45us) server_negotiation.cc:407] Connection header received
0925 02:49:44.229610 (+ 2267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.229637 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.230155 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.230661 (+ 506us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.232393 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.233931 (+ 1538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.235777 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.236516 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.236746 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.242184 (+ 5438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:44.242203 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:44.242220 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:44.242282 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:44.245728 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.246574 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.246581 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.246588 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.246745 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.247132 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.247141 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.247144 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.247651 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:44.247979 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.248389 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.248806 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":352,"threads_started":1}
I20260925 02:49:44.254525 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.234657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36020 (local address 127.5.56.190:35297)
0925 02:49:44.235215 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.235222 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.237835 (+ 2613us) server_negotiation.cc:407] Connection header received
0925 02:49:44.238033 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.238037 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.238103 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.238193 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.239904 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.240781 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.242654 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.243061 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.243175 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.247140 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:44.247147 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:44.247150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:44.247189 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:44.250741 (+ 3552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.251722 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.251726 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.251728 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.251854 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.252384 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.252398 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.252401 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.252843 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:44.253045 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.253700 (+ 655us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.253901 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":279,"threads_started":1}
I20260925 02:49:44.254905 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.234678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.190:35297 (local address 127.0.0.1:36020)
0925 02:49:44.237648 (+ 2970us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:44.237711 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:44.237901 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:44.238304 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:44.238313 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:44.239301 (+ 988us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:44.239736 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:44.239754 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.240968 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.240972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:44.242497 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:44.242519 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.243296 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.243306 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:44.243525 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.244273 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:44.244306 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:44.246889 (+ 2583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:44.250890 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:44.250906 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:44.250914 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:44.251578 (+ 664us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:44.251977 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:44.251982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:44.251986 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:44.252223 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:44.253009 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:44.253018 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:44.253261 (+ 243us) client_negotiation.cc:769] Sending connection context
0925 02:49:44.254206 (+ 945us) client_negotiation.cc:240] Negotiation successful
0925 02:49:44.254524 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2669,"thread_start_us":2478,"threads_started":1}
I20260925 02:49:44.256546 28642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:49:44 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.190@KRBTEST.COM
Sep 25 02:49:44 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304584, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
I20260925 02:49:44.271822 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.257233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.189:44419 (local address 127.0.0.1:42748)
0925 02:49:44.257421 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:44.257441 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:44.257557 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:44.257904 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:44.257908 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:44.258262 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:44.258461 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:44.258470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.259835 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.259841 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:44.260849 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:44.260862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.261592 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.261598 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:44.261782 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.262376 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:44.262453 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:44.265076 (+ 2623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:44.268890 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:44.268896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:44.268901 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:44.269515 (+ 614us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:44.269926 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:44.269931 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:44.269935 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:44.270159 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:44.270792 (+ 633us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:44.270797 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:44.270982 (+ 185us) client_negotiation.cc:769] Sending connection context
0925 02:49:44.271186 (+ 204us) client_negotiation.cc:240] Negotiation successful
0925 02:49:44.271463 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 02:49:44.271885 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.257435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.5.56.189:44419)
0925 02:49:44.257633 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.257641 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.257659 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:49:44.257720 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.257723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.257778 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.257890 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.258604 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.259697 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.261007 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.261423 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.261564 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.265269 (+ 3705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:44.265275 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:44.265280 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:44.265323 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:44.268755 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.269641 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.269645 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.269649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.269796 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.270286 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.270290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.270294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.270663 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:44.270797 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.271281 (+ 484us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.271500 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:49:44.271919 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.253151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36036 (local address 127.5.56.190:35297)
0925 02:49:44.253795 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.253800 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.253818 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:49:44.253857 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.253860 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.253912 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.253992 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.255927 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.256946 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.258734 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.259411 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.259532 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.264928 (+ 5396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:44.264933 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:44.264936 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:44.264968 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:44.268523 (+ 3555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.269520 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.269523 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.269526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.269687 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.270176 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.270180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.270182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.270604 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:44.270879 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.271278 (+ 399us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.271509 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":533,"thread_start_us":321,"threads_started":1}
I20260925 02:49:44.273610 28642 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 02:49:44 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304584, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.188@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
Sep 25 02:49:44 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304582, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.5.56.189@KRBTEST.COM
I20260925 02:49:44.286121 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.274340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52694)
0925 02:49:44.274563 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:44.274581 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:44.274679 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:44.274949 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:44.274955 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:44.275424 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:44.275619 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:44.275630 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.276526 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.276534 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:44.277362 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:44.277372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.278087 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.278092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:44.278288 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.278909 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:44.278925 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:44.280775 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:44.283664 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:44.283669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:44.283673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:44.284199 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:44.284544 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:44.284551 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:44.284556 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:44.284723 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:44.285221 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:44.285230 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:44.285406 (+ 176us) client_negotiation.cc:769] Sending connection context
0925 02:49:44.285567 (+ 161us) client_negotiation.cc:240] Negotiation successful
0925 02:49:44.285842 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":1}
I20260925 02:49:44.286630 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.274562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52694 (local address 127.5.56.188:36105)
0925 02:49:44.274714 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.274718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.274731 (+ 13us) server_negotiation.cc:407] Connection header received
0925 02:49:44.274765 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.274768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.274816 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.274886 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.275760 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.276407 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.277488 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.277918 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.278073 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.280941 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:44.280946 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:44.280951 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:44.280987 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:44.283532 (+ 2545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.284325 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.284328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.284331 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.284435 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.284837 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.284840 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.284842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.285100 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:44.285216 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.285696 (+ 480us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.286030 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":70,"server-negotiator.queue_time_us":45}
I20260925 02:49:44.288368 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.273720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.5.56.189:44419)
0925 02:49:44.273949 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.273955 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.273974 (+ 19us) server_negotiation.cc:407] Connection header received
0925 02:49:44.274024 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.274029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.274123 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.274222 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.275855 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.276533 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.277736 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.278178 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.278315 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.282525 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:44.282533 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:44.282538 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:44.282578 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:44.285522 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.286335 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.286342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.286348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.286543 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.287025 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.287030 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.287033 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.287311 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:44.287525 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.287894 (+ 369us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.288054 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260925 02:49:44.291374 28494 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } attempt: 4
I20260925 02:49:44.293009 28642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783: Bootstrap starting.
I20260925 02:49:44.293874 5346 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 02:49:44.297124 28642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:44.298206 28642 log.cc:824] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783: Log is configured to *not* fsync() on all Append() calls
Sep 25 02:49:44 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304583, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.190@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
I20260925 02:49:44.296974 28468 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1b588414e483452cb7b115e0d2f214bf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36036
I20260925 02:49:44.301287 28642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783: No bootstrap required, opened a new log
I20260925 02:49:44.307374 28497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.292015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52710)
0925 02:49:44.292199 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:44.292221 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:44.292336 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:44.292893 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:44.292898 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:44.293354 (+ 456us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:44.293510 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:44.293519 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.294680 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.294687 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:44.295588 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:44.295598 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.296331 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.296337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:44.296535 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.297201 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:44.297229 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:44.300288 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:44.304489 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:44.304509 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:44.304515 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:44.305029 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:44.305450 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:44.305455 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:44.305458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:44.305613 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:44.306273 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:44.306278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:44.306478 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 02:49:44.306634 (+ 156us) client_negotiation.cc:240] Negotiation successful
0925 02:49:44.306963 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260925 02:49:44.307382 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.292233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52710 (local address 127.5.56.188:36105)
0925 02:49:44.292593 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.292599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.292615 (+ 16us) server_negotiation.cc:407] Connection header received
0925 02:49:44.292671 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.292676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.292748 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.292862 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.293643 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.294545 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.295729 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.296189 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.296317 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.300498 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:44.300504 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:44.300509 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:44.300544 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:44.304326 (+ 3782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.305153 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.305159 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.305162 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.305327 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.305729 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.305734 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.305737 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.306124 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:44.306315 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.306761 (+ 446us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.307001 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 02:49:44.312532 28642 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } }
I20260925 02:49:44.313051 28642 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ba095a0af7a42d4961a10c8a4cda783, State: Initialized, Role: FOLLOWER
I20260925 02:49:44.313550 28494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf: Bootstrap starting.
I20260925 02:49:44.313555 28642 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } }
I20260925 02:49:44.314570 28650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } } }
I20260925 02:49:44.314956 28650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:44.315914 28642 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:44.317858 28494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:44.319559 28494 log.cc:824] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:44.323679 28494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf: No bootstrap required, opened a new log
I20260925 02:49:44.332082 28494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } }
I20260925 02:49:44.332547 28494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b588414e483452cb7b115e0d2f214bf, State: Initialized, Role: FOLLOWER
W20260925 02:49:44.332598 28663 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 02:49:44.332738 28663 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 02:49:44.333060 28494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } }
I20260925 02:49:44.334228 28665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } } }
I20260925 02:49:44.334933 28665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:44.335467 28494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:44.335556 28468 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 02:49:44.335736 28468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:44.339107 28468 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } }
I20260925 02:49:44.340554 28468 leader_election.cc:288] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [CANDIDATE]: Term 1 election: Requested vote from peers e979d7e9a61a4c1fab4f123145fe8f28 (127.5.56.189:44419), 0ba095a0af7a42d4961a10c8a4cda783 (127.5.56.188:36105)
I20260925 02:49:44.342015 28617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b588414e483452cb7b115e0d2f214bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0ba095a0af7a42d4961a10c8a4cda783"
I20260925 02:49:44.342516 28617 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:44.345994 28617 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b588414e483452cb7b115e0d2f214bf in term 1.
I20260925 02:49:44.346566 28444 leader_election.cc:302] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ba095a0af7a42d4961a10c8a4cda783, 1b588414e483452cb7b115e0d2f214bf; no voters:
I20260925 02:49:44.347226 28665 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [term 1 FOLLOWER]: Leader election won for term 1
I20260925 02:49:44.356103 28665 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [term 1 LEADER]: Becoming Leader. State: Replica: 1b588414e483452cb7b115e0d2f214bf, State: Running, Role: LEADER
I20260925 02:49:44.330150 28575 master_runner.cc:186] Error getting master registration for 127.5.56.190:35297: OK, instance_id { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" instance_seqno: 1790304583147236 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:49:44.356557 28575 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:49:44.357684 28665 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } }
I20260925 02:49:44.375144 28668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b588414e483452cb7b115e0d2f214bf. Latest consensus state: current_term: 1 leader_uuid: "1b588414e483452cb7b115e0d2f214bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } } }
I20260925 02:49:44.375710 28668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [sys.catalog]: This master's current role is: LEADER
W20260925 02:49:44.386521 28447 tablet.cc:2369] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:49:44.394629 28568 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } attempt: 5
I20260925 02:49:44.397084 28676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 02:49:44.422703 28648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.374129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36042 (local address 127.5.56.190:35297)
0925 02:49:44.374339 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.374355 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.381872 (+ 7517us) server_negotiation.cc:407] Connection header received
0925 02:49:44.383271 (+ 1399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.383277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.383465 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.383590 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.388046 (+ 4456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.389227 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.404327 (+ 15100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.405352 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.405497 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.409614 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:44.409621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:44.409626 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:44.409681 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:44.415167 (+ 5486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.418983 (+ 3816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.418990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.418993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.419286 (+ 293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.420034 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.420039 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.420074 (+ 35us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.420626 (+ 552us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:44.421851 (+ 1225us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.422033 (+ 182us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.422273 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":51}
I20260925 02:49:44.426927 28676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 02:49:44.428002 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.382500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.5.56.189:44419)
0925 02:49:44.382664 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.382670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.386818 (+ 4148us) server_negotiation.cc:407] Connection header received
0925 02:49:44.388309 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.388315 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.388405 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.388474 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.393143 (+ 4669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.394250 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.404534 (+ 10284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.405520 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.405646 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.413721 (+ 8075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:44.413731 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:44.413739 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:44.413802 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:44.418068 (+ 4266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.423310 (+ 5242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.423316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.423319 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.423490 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.423926 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.423955 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.423959 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.424503 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:44.424910 (+ 407us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.426853 (+ 1943us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.427327 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 02:49:44.439495 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.387425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52714 (local address 127.5.56.188:36105)
0925 02:49:44.387621 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.387629 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.407077 (+ 19448us) server_negotiation.cc:407] Connection header received
0925 02:49:44.410698 (+ 3621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.410707 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.410820 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.410920 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.423411 (+ 12491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.424513 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.426404 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.427099 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.427239 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.432172 (+ 4933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:44.432178 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:44.432182 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:44.432223 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:44.435946 (+ 3723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.436972 (+ 1026us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.436979 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.436982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.437147 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.437615 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.437620 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.437623 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.437957 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:44.438223 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.438855 (+ 632us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.439092 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
Sep 25 02:49:44 dist-test-slave-n563 krb5kdc[28413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790304583, etypes {rep=17 tkt=17 ses=17}, kudu/127.5.56.189@KRBTEST.COM for kudu/127.5.56.188@KRBTEST.COM
W20260925 02:49:44.448997 28596 tablet.cc:2369] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 02:49:44.450526 28617 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [term 1 FOLLOWER]: Refusing update from remote peer 1b588414e483452cb7b115e0d2f214bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:49:44.451644 28665 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:49:44.446918 28426 master_runner.cc:186] Error getting master registration for 127.5.56.189:44419: OK, instance_id { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" instance_seqno: 1790304583667583 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 02:49:44.462524 28426 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 02:49:44.464473 28570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.395343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.5.56.188:36105 (local address 127.0.0.1:52722)
0925 02:49:44.395770 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0925 02:49:44.395793 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0925 02:49:44.395942 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 02:49:44.429695 (+ 33753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 02:49:44.429700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 02:49:44.430233 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 02:49:44.430475 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:44.430488 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.432351 (+ 1863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.432358 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:44.433830 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 02:49:44.433855 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.439806 (+ 5951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.439813 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 02:49:44.440060 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.440833 (+ 773us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 02:49:44.440871 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0925 02:49:44.443403 (+ 2532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 02:49:44.459526 (+ 16123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:44.459532 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:44.459536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:44.460143 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:44.460737 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 02:49:44.460742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 02:49:44.460773 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0925 02:49:44.461108 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 02:49:44.462436 (+ 1328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 02:49:44.462442 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 02:49:44.462777 (+ 335us) client_negotiation.cc:769] Sending connection context
0925 02:49:44.463059 (+ 282us) client_negotiation.cc:240] Negotiation successful
0925 02:49:44.463764 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276}
I20260925 02:49:44.466637 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.396207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52722 (local address 127.5.56.188:36105)
0925 02:49:44.429224 (+ 33017us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.429229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.429255 (+ 26us) server_negotiation.cc:407] Connection header received
0925 02:49:44.429306 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.429311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.429411 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.430672 (+ 1261us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.430971 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.432187 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.434050 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.434870 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.444265 (+ 9395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.444489 (+ 224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 02:49:44.444495 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 02:49:44.444500 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 02:49:44.444537 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0925 02:49:44.458826 (+ 14289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.460337 (+ 1511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.460344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.460360 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.460541 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 02:49:44.461348 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 02:49:44.461389 (+ 41us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 02:49:44.461395 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0925 02:49:44.462166 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 02:49:44.464908 (+ 2742us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.465181 (+ 273us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.465643 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32878,"thread_start_us":33250,"threads_started":1}
I20260925 02:49:44.472729 28568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28: Bootstrap starting.
I20260925 02:49:44.472892 28650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b588414e483452cb7b115e0d2f214bf. Latest consensus state: current_term: 1 leader_uuid: "1b588414e483452cb7b115e0d2f214bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } } }
I20260925 02:49:44.473181 28650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:44.486285 28691 mvcc.cc:205] Tried to move back new op lower bound from 7333087577895981056 to 7333087577543999488. Current Snapshot: MvccSnapshot[applied={T|T < 7333087577895981056}]
I20260925 02:49:44.486721 28568 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28: Neither blocks nor log segments found. Creating new log.
I20260925 02:49:44.488874 28568 log.cc:824] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28: Log is configured to *not* fsync() on all Append() calls
I20260925 02:49:44.491573 28665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b588414e483452cb7b115e0d2f214bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } } }
I20260925 02:49:44.491897 28665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:44.493341 28650 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b588414e483452cb7b115e0d2f214bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } } }
I20260925 02:49:44.493589 28650 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:44.494704 28676 catalog_manager.cc:1347] Generated new cluster ID: f80bdeadd35d4704bf7ed934a614b3c1
I20260925 02:49:44.494807 28676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 02:49:44.501693 28568 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28: No bootstrap required, opened a new log
I20260925 02:49:44.510336 28568 raft_consensus.cc:348] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } }
I20260925 02:49:44.510926 28568 raft_consensus.cc:729] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e979d7e9a61a4c1fab4f123145fe8f28, State: Initialized, Role: FOLLOWER
I20260925 02:49:44.511507 28568 consensus_queue.cc:260] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } }
I20260925 02:49:44.525286 28696 sys_catalog.cc:455] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } } }
I20260925 02:49:44.525882 28696 sys_catalog.cc:458] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:44.526842 28568 sys_catalog.cc:564] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 02:49:44.526938 28543 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1b588414e483452cb7b115e0d2f214bf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e979d7e9a61a4c1fab4f123145fe8f28"
I20260925 02:49:44.527424 28543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [term 0 FOLLOWER]: Advancing to term 1
I20260925 02:49:44.531050 28543 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b588414e483452cb7b115e0d2f214bf in term 1.
I20260925 02:49:44.532018 28542 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [term 1 FOLLOWER]: Refusing update from remote peer 1b588414e483452cb7b115e0d2f214bf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 02:49:44.532892 28668 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 02:49:44.540112 28676 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 02:49:44.541985 28676 catalog_manager.cc:1504] Loading token signing keys...
I20260925 02:49:44.559114 28676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf: Generated new TSK 0
I20260925 02:49:44.560398 28676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 02:49:44.607981 28700 sys_catalog.cc:455] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1b588414e483452cb7b115e0d2f214bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } } }
I20260925 02:49:44.608443 28700 sys_catalog.cc:458] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:44.608886 28696 sys_catalog.cc:455] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b588414e483452cb7b115e0d2f214bf. Latest consensus state: current_term: 1 leader_uuid: "1b588414e483452cb7b115e0d2f214bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } } }
I20260925 02:49:44.609163 28696 sys_catalog.cc:458] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 02:49:44.609020 28668 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1b588414e483452cb7b115e0d2f214bf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b588414e483452cb7b115e0d2f214bf" member_type: VOTER last_known_addr { host: "127.5.56.190" port: 35297 } } peers { permanent_uuid: "e979d7e9a61a4c1fab4f123145fe8f28" member_type: VOTER last_known_addr { host: "127.5.56.189" port: 44419 } } peers { permanent_uuid: "0ba095a0af7a42d4961a10c8a4cda783" member_type: VOTER last_known_addr { host: "127.5.56.188" port: 36105 } } }
I20260925 02:49:44.609303 28668 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b588414e483452cb7b115e0d2f214bf [sys.catalog]: This master's current role is: LEADER
I20260925 02:49:44.658227 28714 catalog_manager.cc:1259] Loaded cluster ID: f80bdeadd35d4704bf7ed934a614b3c1
I20260925 02:49:44.658344 28714 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28: loading cluster ID for follower catalog manager: success
I20260925 02:49:44.662228 28714 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28: acquiring CA information for follower catalog manager: success
I20260925 02:49:44.675341 28714 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e979d7e9a61a4c1fab4f123145fe8f28: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 02:49:44.694200 28648 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:44.679977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36054 (local address 127.5.56.190:35297)
0925 02:49:44.680192 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.680199 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.691232 (+ 11033us) server_negotiation.cc:407] Connection header received
0925 02:49:44.691660 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.691670 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.691824 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.691969 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.693757 (+ 1788us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36054: BlockingRecv error: recv got EOF from 127.0.0.1:36054 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260925 02:49:44.701690 28572 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:44.690703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42794 (local address 127.5.56.189:44419)
0925 02:49:44.690953 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.690961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.697884 (+ 6923us) server_negotiation.cc:407] Connection header received
0925 02:49:44.699224 (+ 1340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.699231 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.699346 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.699427 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.701405 (+ 1978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42794: BlockingRecv error: recv got EOF from 127.0.0.1:42794 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260925 02:49:44.704720 28682 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:44.698469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52732 (local address 127.5.56.188:36105)
0925 02:49:44.698675 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.698681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.702485 (+ 3804us) server_negotiation.cc:407] Connection header received
0925 02:49:44.702794 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.702801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.702935 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.703041 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.704364 (+ 1323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52732: BlockingRecv error: recv got EOF from 127.0.0.1:52732 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 02:49:44.757532 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.740683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42800 (local address 127.5.56.189:44419)
0925 02:49:44.740901 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.740909 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.748523 (+ 7614us) server_negotiation.cc:407] Connection header received
0925 02:49:44.749155 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.749163 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.749433 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.749595 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:44.750522 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.751719 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.753943 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.754769 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.754888 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.755764 (+ 876us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:44.756559 (+ 795us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:44.756719 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.756999 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.757114 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260925 02:49:44.801746 28682 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:44.791452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52736 (local address 127.5.56.188:36105)
0925 02:49:44.791678 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.791685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.799311 (+ 7626us) server_negotiation.cc:407] Connection header received
0925 02:49:44.799966 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.799974 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.800116 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.800218 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.801417 (+ 1199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52736: BlockingRecv error: recv got EOF from 127.0.0.1:52736 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:49:44.849722 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.834664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.5.56.189:44419)
0925 02:49:44.834869 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.834877 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.842340 (+ 7463us) server_negotiation.cc:407] Connection header received
0925 02:49:44.842976 (+ 636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.842982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.843107 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.843180 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:44.843901 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.845136 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.847300 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.848000 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.848108 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.848769 (+ 661us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:44.848940 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:44.849010 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.849252 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.849355 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260925 02:49:44.893558 28682 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:44.883602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.5.56.188:36105)
0925 02:49:44.883834 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.883840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.891422 (+ 7582us) server_negotiation.cc:407] Connection header received
0925 02:49:44.891971 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.891977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.892086 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.892172 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.893327 (+ 1155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52750: BlockingRecv error: recv got EOF from 127.0.0.1:52750 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
I20260925 02:49:44.946465 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:44.931360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42814 (local address 127.5.56.189:44419)
0925 02:49:44.931573 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.931580 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.939187 (+ 7607us) server_negotiation.cc:407] Connection header received
0925 02:49:44.939732 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.939738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.939843 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.939922 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:44.940683 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.941884 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.944004 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:44.944716 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:44.944847 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:44.945557 (+ 710us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:44.945723 (+ 166us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:44.945785 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:44.946007 (+ 222us) server_negotiation.cc:299] Negotiation successful
0925 02:49:44.946106 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260925 02:49:44.990677 28682 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:44.980594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52760 (local address 127.5.56.188:36105)
0925 02:49:44.980820 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:44.980827 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:44.988361 (+ 7534us) server_negotiation.cc:407] Connection header received
0925 02:49:44.988955 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:44.988963 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:44.989099 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:44.989188 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:44.990431 (+ 1243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52760: BlockingRecv error: recv got EOF from 127.0.0.1:52760 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260925 02:49:45.045931 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:45.030705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42828 (local address 127.5.56.189:44419)
0925 02:49:45.030963 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.030970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.038532 (+ 7562us) server_negotiation.cc:407] Connection header received
0925 02:49:45.039097 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.039108 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.039206 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.039286 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:45.040092 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.041335 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.043429 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.044171 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.044290 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:45.045009 (+ 719us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:45.045166 (+ 157us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:45.045238 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:45.045490 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 02:49:45.045588 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
W20260925 02:49:45.084775 28682 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:45.079908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52768 (local address 127.5.56.188:36105)
0925 02:49:45.080184 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.080191 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.083001 (+ 2810us) server_negotiation.cc:407] Connection header received
0925 02:49:45.083241 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.083247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.083365 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.083497 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:45.084490 (+ 993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52768: BlockingRecv error: recv got EOF from 127.0.0.1:52768 (error 108)
Metrics: {"server-negotiator.queue_time_us":130}
I20260925 02:49:45.103915 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:45.096359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42836 (local address 127.5.56.189:44419)
0925 02:49:45.096589 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.096596 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.097119 (+ 523us) server_negotiation.cc:407] Connection header received
0925 02:49:45.097340 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.097345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.097440 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.097557 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:45.098040 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.099273 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.101353 (+ 2080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.102082 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.102223 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:45.103006 (+ 783us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:45.103195 (+ 189us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:45.103289 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:45.103476 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 02:49:45.103564 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
W20260925 02:49:45.124775 28682 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:45.120023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52782 (local address 127.5.56.188:36105)
0925 02:49:45.120210 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.120216 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.123072 (+ 2856us) server_negotiation.cc:407] Connection header received
0925 02:49:45.123285 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.123292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.123395 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.123518 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:45.124505 (+ 987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52782: BlockingRecv error: recv got EOF from 127.0.0.1:52782 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:49:45.151773 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:45.144295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42840 (local address 127.5.56.189:44419)
0925 02:49:45.144510 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.144517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.145084 (+ 567us) server_negotiation.cc:407] Connection header received
0925 02:49:45.145313 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.145320 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.145428 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.145536 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:45.146033 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.147315 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.149288 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.149991 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.150143 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:45.150672 (+ 529us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:45.150871 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:45.150974 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:45.151234 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 02:49:45.151353 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260925 02:49:45.170910 28682 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:45.168812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52798 (local address 127.5.56.188:36105)
0925 02:49:45.169016 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.169023 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.169434 (+ 411us) server_negotiation.cc:407] Connection header received
0925 02:49:45.169660 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.169668 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.169750 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.169860 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:45.170679 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52798: BlockingRecv error: recv got EOF from 127.0.0.1:52798 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 02:49:45.212630 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:45.206511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.5.56.189:44419)
0925 02:49:45.206710 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.206717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.207140 (+ 423us) server_negotiation.cc:407] Connection header received
0925 02:49:45.207314 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.207320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.207428 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.207537 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:45.207944 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.209054 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.210507 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.211234 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.211381 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:45.211859 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:45.212000 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:45.212094 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:45.212274 (+ 180us) server_negotiation.cc:299] Negotiation successful
0925 02:49:45.212344 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 02:49:45.230197 28682 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:45.225726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52804 (local address 127.5.56.188:36105)
0925 02:49:45.225968 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.225976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.228054 (+ 2078us) server_negotiation.cc:407] Connection header received
0925 02:49:45.228398 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.228404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.228508 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.228630 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:45.229867 (+ 1237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52804: BlockingRecv error: recv got EOF from 127.0.0.1:52804 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 02:49:45.303195 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:45.298245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42854 (local address 127.5.56.189:44419)
0925 02:49:45.298534 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.298543 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.298729 (+ 186us) server_negotiation.cc:407] Connection header received
0925 02:49:45.298938 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.298942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.299015 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.299094 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:45.299471 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.300199 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.301533 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.301966 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.302057 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:45.302445 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:45.302559 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:45.302626 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:45.302777 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 02:49:45.302842 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
W20260925 02:49:45.310832 28682 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 02:49:45.309012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52810 (local address 127.5.56.188:36105)
0925 02:49:45.309197 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.309201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.309581 (+ 380us) server_negotiation.cc:407] Connection header received
0925 02:49:45.309733 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.309737 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.309808 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.309906 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0925 02:49:45.310661 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52810: BlockingRecv error: recv got EOF from 127.0.0.1:52810 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:49:45.334838 28663 catalog_manager.cc:1259] Loaded cluster ID: f80bdeadd35d4704bf7ed934a614b3c1
I20260925 02:49:45.334936 28663 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783: loading cluster ID for follower catalog manager: success
I20260925 02:49:45.337535 28663 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783: acquiring CA information for follower catalog manager: success
I20260925 02:49:45.339409 28663 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0ba095a0af7a42d4961a10c8a4cda783: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 02:49:45.448086 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:45.442817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42864 (local address 127.5.56.189:44419)
0925 02:49:45.443047 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.443055 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.443412 (+ 357us) server_negotiation.cc:407] Connection header received
0925 02:49:45.443649 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.443654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.443725 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.443825 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:45.444301 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.445002 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.446371 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.446814 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.446951 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:45.447368 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:45.447479 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:45.447579 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:45.447735 (+ 156us) server_negotiation.cc:299] Negotiation successful
0925 02:49:45.447814 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260925 02:49:45.460355 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:45.454279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52824 (local address 127.5.56.188:36105)
0925 02:49:45.454528 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.454535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.454902 (+ 367us) server_negotiation.cc:407] Connection header received
0925 02:49:45.455126 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.455130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.455290 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.455415 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:45.455885 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.456618 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.457959 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.458442 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.458623 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:45.459170 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:45.459756 (+ 586us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:45.459926 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:45.459978 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 02:49:45.460061 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260925 02:49:45.473801 28819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:45.466222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36070 (local address 127.5.56.190:35297)
0925 02:49:45.466792 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.466798 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.466816 (+ 18us) server_negotiation.cc:407] Connection header received
0925 02:49:45.467061 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.467068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.467309 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.468072 (+ 763us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:45.468243 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.469407 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.471283 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.471684 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.471827 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:45.472408 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:45.472757 (+ 349us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:45.473273 (+ 516us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:45.473328 (+ 55us) server_negotiation.cc:299] Negotiation successful
0925 02:49:45.473424 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":316,"threads_started":1}
I20260925 02:49:45.474092 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:45.466928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42878 (local address 127.5.56.189:44419)
0925 02:49:45.467165 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.467173 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.467661 (+ 488us) server_negotiation.cc:407] Connection header received
0925 02:49:45.467958 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.467967 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.468065 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.468165 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:45.468987 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.469884 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.472061 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.472732 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.472857 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:45.473376 (+ 519us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:45.473491 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:45.473573 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:45.473750 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 02:49:45.473829 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260925 02:49:45.474803 28682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 02:49:45.467695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.5.56.188:36105)
0925 02:49:45.467931 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0925 02:49:45.467939 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 02:49:45.468533 (+ 594us) server_negotiation.cc:407] Connection header received
0925 02:49:45.468711 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 02:49:45.468716 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 02:49:45.468790 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 02:49:45.468883 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 02:49:45.469322 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.470364 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.472430 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 02:49:45.473049 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 02:49:45.473177 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 02:49:45.473633 (+ 456us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 02:49:45.473760 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 02:49:45.473849 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 02:49:45.474441 (+ 592us) server_negotiation.cc:299] Negotiation successful
0925 02:49:45.474518 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260925 02:49:45.483127 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 28426
I20260925 02:49:45.484122 28488 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:45.595814 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 28426
W20260925 02:49:45.618095 28591 connection.cc:535] server connection from 127.0.0.1:52710 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 02:49:45.618876 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 28501
I20260925 02:49:45.619980 28563 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:45.730172 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 28501
I20260925 02:49:45.748720 5346 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task87C_4R/build/asan/bin/kudu pid 28575
I20260925 02:49:45.749853 28637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260925 02:49:45.857932 5346 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task87C_4R/build/asan/bin/kudu with pid 28575
2026-09-25T02:49:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4554 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9082 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (373054 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST